Bulk Writer is a small library which facilitates building fast, pull-based ETL processes in C# using SqlBulkCopy.
Documentation can be found at https://jbogard.github.io/bulk-writer/
Bulk Writer is available on NuGet and can be installed using the package manager console:
PM> Install-Package BulkWriter
var q =
from entity in GetAllEntities()
where entity.IsActive && SomeOtherPredicate(entity)
from zipCode in GetAllZipCodes()
where zipCode.IsInContiguousStates && SomeOtherPredicate(zipCode)
let distance = GetDistance(entity, zipCode)
let arbitraryData = CreateSomeArbitraryData(entity, zipCode)
where distance > 0
select new EntityToZipCodeDistance {
EntityId = entity.Id,
ZipCode = zipCode.Zip,
Distance = distance,
ArbitraryData = arbitraryData
};
using (var bulkWriter = new BulkWriter<EntityToZipCodeDistance>(connectionString))
{
bulkWriter.WriteToDatabase(q);
}
// or async
using (var bulkWriter = new BulkWriter<EntityToZipCodeDistance>(connectionString))
{
await bulkWriter.WriteToDatabaseAsync(q);
}
// or async enumerables with .NET Standard 2.1 or later
var u = q.ToAsyncEnumerable(); //
using (var bulkWriter = new BulkWriter<EntityToZipCodeDistance>(connectionString))
{
await bulkWriter.WriteToDatabaseAsync(u);
}Run the following command once to setup your environment.
PS> .\setup.ps1
Run the command below to build and test the project.
PS> .\psake.cmd
Pull Requests are welcome. If you identify a bug or would like to make a feature request feel free to submit a GitHub Issue to start a discussion.