Data Engineering Zoomcamp

There was a problem with Q3 of homework 1 - we updated the possible answers

We're starting the first module!

In this module, we will learn about Docker and Terraform, and also refresh our SQL knowledge

Module materials: https://github.com/DataTalksClub/data-engineering-zoomcamp/tree/main/01-docker-terraform

Homework: https://github.com/DataTalksClub/data-engineering-zoomcamp/blob/main/cohorts/2025/01-docker-terraform/homework.md

Have fun!

We're starting!

Join the stream:

https://www.youtube.com/watch?v=X8cEEwi8DTM

(Also available later in recording)

Ask questions: https://app.sli.do/event/aFZgu1o3nbgJTyX1AH1dtF/live/questions

We're starting in one hour!

We will stream on our YouTube channel:

https://youtube.com/c/datatalksclub

You can already subscribe to it. We will share the link to the stream later

Use this link for asking questions: https://app.sli.do/event/aFZgu1o3nbgJTyX1AH1dtF/live/questions

See you soon!

Are you ready for the course?

The launch stream is tomorrow (Monday) at 17:00 CET

You can ask your questions here in advance:

https://app.sli.do/event/aFZgu1o3nbgJTyX1AH1dtF/live/questions

See you tomorrow!

Starting now!

Stream: https://www.youtube.com/watch?v=DPnAOu2csYA
Questions: https://app.sli.do/event/aFZgu1o3nbgJTyX1AH1dtF

You can join now or watch later in recording

You can ask your questions here: https://app.sli.do/event/aFZgu1o3nbgJTyX1AH1dtF/live/questions

We will start the Q&A stream in 1 hour 45 minutes. Sign up here: https://lu.ma/e7yc4uyb

https://www.youtube.com/watch?v=R0JAFvDCmSY

This is a demo of Kesta - the tool we will use in the next edition of DE Zoomcamp

We will use Kestra as the orchestrator in module 2 in the new edition of the course

Kesta is a great tool with intuitive UI and pleasant user experience

They also support our course and our community and we are very thankful for that!

Right now they're launching on Product Hunt, so let's also support them and give them an upvote:

https://www.producthunt.com/posts/kestra

Hi everyone!

We want to interview our course graduates and to make a series of podcasts/articles about your journey in our community

- How did our courses impact your journey?
- What did you gain from it?
- What was your career path like after completing the course?

Would you be interested in sharing your story?

Fill in the form and we will contact you
https://airtable.com/appY3dCxs3rPhB60s/shrWvYKzErRJtVTPj

Your experience will be highly motivational for the future students!

If you're interested in learning more about dlt, we will have another workshop with them as a part of our LLM Zoomcamp

📢 Join us for the "Open source data ingestion for RAGs with dlt" Event!
🎙️ Speaker: Akela Drissner
🗓️ Date: Monday, July 8
🕟 Time: 4:30 PM (CET)

In this hands-on workshop, we’ll learn how to build a data ingestion pipeline using dlt to load data from a REST API into LanceDB so you can have an always up-to-date RAG.


Register here: https://lu.ma/cnpdoc5n

(You don't have to be enrolled in the course to take the workshop)

Hello everyone!

We finally published the public leaderboard with top-100 participants for the 2024 edition

Thanks for being a part of the course!

https://github.com/DataTalksClub/data-engineering-zoomcamp/blob/main/cohorts/2024/leaderboard.md

We have updated the names on your certificates. If you haven't changed the certificate name, sorry, but we won't regenerate certificates anymore

Also unfortunately there was a mistake in scoring the second project batch. We had to recall 6 certificates for people who didn't pass the project. Apologies for the error on our side.

We're starting a stream on getting your first data engineering job now:

https://www.youtube.com/watch?v=1AchMU9xf0Q

Join now or watch later in replay

Congratulations to the top-100 leaderboard! You worked really hard to get there!

We will publish your names publicly on the course github repo - like here https://github.com/DataTalksClub/machine-learning-zoomcamp/blob/master/cohorts/2023/leaderboard.md

To do it, fill in this form - https://courses.datatalks.club/de-zoomcamp-2024/homework/leaderboard

This is also your last chance to update the certificate name. We will regenerate certificates one more time in one week and after that we will not do it again.

We have closed the evaluation form - now the course has officially ended! Yay!

Congratulations to everyone who took part - especially the 359 people who graduated. It was tough yet you made it - great work!

You can get your certificate using the instructions from here https://github.com/DataTalksClub/data-engineering-zoomcamp/blob/main/certificates.md

If your certificate previously had leaderboard display name there, but you updated the certiciate name - now the pdf should be updated.

If you didn't update your certificate name - it seems you'll need to update your legal name now. Sorry, Nervous Jang, Confident Brown, Competent Brown, Inspiring Fermi, Data Engineering Zoomcamp 2024 and others.

Don't forget to evaluate your peers! The deadline is over, but you have a few more hours to do it if you have forgotten. We'll close the form tonight

A reminder:

You need to update the "certificate name" field for your certificate, otherwise we will have to use your leaderboard display name

You can do it here

https://courses.datatalks.club/de-zoomcamp-2024/enrollment

(You have to be logged in)

We don't have the bandwidth to individually re-issue certificates if you forget to do it, so make sure you do it before the deadline for the second attempt is over

If you passed the first project attempt, your certificate should be available!

https://certificate.datatalks.club/dezoomcamp/2024/fe629854d45c559e9c10b3b8458ea392fdeb68a9.pdf

For now, use this instruction to get the certificate: https://github.com/DataTalksClub/data-engineering-zoomcamp/blob/main/certificates.md

If you didn't specify the name for the certificate, we used your leaderboard display name (sorry, Compassionate Perlman, Awesome Cray, Frosty Ellis and others)

Also, the peer review assignments for attempt 2 are now available at https://courses.datatalks.club/de-zoomcamp-2024/project/project2/eval

Enjoy learning from your peers and thanks everyone for your hard work!

We started preparing the certificates for those who passed the first project attempt (it's around 150 people! great work!)

Please make sure the name on the certificate is the correct one

Go to https://courses.datatalks.club/de-zoomcamp-2024/enrollment and update it (you need to be logged in)

We finally scored the first project attempt

You can see the results here: https://courses.datatalks.club/de-zoomcamp-2024/project/project1/results

We still need to implement a few things, like displaying the feedback from your peers, but at least now you can see if you passed or failed

Also, the deadline for attempt 2 is extended till Thursday, 18 of April. Have fun!

Now it's time to evaluate your peers!

You can find the assignments here:

https://courses.datatalks.club/de-zoomcamp-2024/project/project1/eval

Make sure you're logged in

We're extending the deadline for submitting the project attempt 1 till Wednesday 23:00. Use this time wisely.


We also opened the form for submitting attempt 2

https://courses.datatalks.club/de-zoomcamp-2024/project/project2


Have fun!

Hi everyone!

How is it going with the project?

Remember that awesome-data-engineering document we mentioned at the launch stream but never actually used?

We now put it to github:

https://github.com/DataTalksClub/data-engineering-zoomcamp/blob/main/awesome-data-engineering.md

Hopefully it's helpful! PRs are welcome as always

We're starting office hours in 5 minutes

We will talk about your projects

Questions: https://app.sli.do/event/su9wCLiM9nHnCwtGBfgicX

Stream: https://www.youtube.com/watch?v=Of0ZqLrZXmc (recorded, available for replay later)

Projects: https://github.com/DataTalksClub/data-engineering-zoomcamp/blob/main/cohorts/2024/project.md

We're starting the webinar about starting a career in data engineering

https://www.youtube.com/watch?v=xxSu2y7q9cg

(The recording will stay on the channel and is available for watching later)

The streaming homework is ready

https://github.com/DataTalksClub/data-engineering-zoomcamp/blob/main/cohorts/2024/06-streaming/homework.md

It took some time for us to prepare it, and it messed up with your plans, so you will have quite a bit of time to complete it (the deadline will overlap with projects). We will announce the deadlines later, but if you're now on vacations, you should be able to pick it up when you come back.

We will publish the form soon too.

Also, we updated the python videos from the streaming module - now they have much better audio quality

Thanks Ella Sahnan and Artsiom Turevich for help with the homework and the videos.

Finally, next week, we should start working on the projects. So on Monday, at 16:30 CET, we will have office hours to discuss it. You can already ask your questions here: https://app.sli.do/event/su9wCLiM9nHnCwtGBfgicX

Hi everyone!

We want to make homework 6 more interesting and helpful for you, but we need a bit more time to prepare it. Through this homework, you will learn how to use PySpark Streaming with Kafka. We will publish it by Monday.

You can already start with Module 6 videos. There are Java videos that focus more on the theory as well as Python videos that focus more on practice. We didn't have a chance to re-record the Python ones this year, but let us know if you want to volunteer and help with it.

Link to the module: https://github.com/DataTalksClub/data-engineering-zoomcamp/tree/main/06-streaming

In the meantime, you can work on the 2nd workshop and its homework.