![]() ![]() What kind of reliability do you need? It's tough to beat the fault tolerance of a cloud-native service.Will it integrate with new data sources - even those that are less popular or custom-built? To be prepared to take advantage of new data sources, consider an extensible solution like Stitch, whose Singer open source framework lets you integrate new data sources.Does an ETL tool integrate with every data source you use today?.How do you pick the most suitable ETL tool for your business? Start by asking questions specific to your requirements: They support integrations with non-AWS data sources through graphical interfaces, and offer attractive pricing models. Third-party AWS ETL tools often have advantages over AWS Glue and internal pipelines. If you need to include other sources in your ETL plan, a third-party ETL tool is a better choice. PostgreSQL in Amazon VPC running on EC2.Microsoft SQL Server in Amazon VPC running on EC2.However, Glue supports only services running on AWS: Write metadata pertaining to the ETL job into the AWS Glue Data Catalog.Transform data based on code generated automatically by AWS Glue.Set up a schedule or identify events to trigger an ETL job.Glue's ETL process is similar to that of a manually coded data pipeline: Glue may be a good choice if you're moving data from an Amazon data source to an Amazon data warehouse. Should you stick with AWS Glue for ETL?ĪWS Glue is a managed ETL service that you control from the AWS Management Console. In the AWS world, AWS Glue can handle ETL jobs, or you can consider a third-party service like Stitch. If you want to follow Magnusson's advice, you can turn to a SaaS service to handle ETL tasks. There is nothing more soul sucking than writing, maintaining, modifying, and supporting ETL to produce data that you yourself never get to use or consume."įortunately, there's a smart alternative to writing and maintaining your own ETL code. For the love of everything sacred and holy in the profession, this should not be a dedicated or specialized role. ![]() As Jeff Magnusson, vice president of Stitch Fix, says, " Engineers should not write ETL. Given all that, many organizations choose to avoid manually coding data pipelines.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |