Hey

Sometimes I use some Google tutorial to do some training. But I like to automate (yes, I know you know!). So, let's talk about CI/CD for data processing in GCP. I'm going to use this tutorial:

In summary:

This tutorial describes how to set up a continuous integration/continuous deployment (CI/CD) pipeline for processing data by implementing CI/CD methods with managed products on Google Cloud. Data scientists and analysts can adapt the methodologies from CI/CD practices to help to ensure high quality, maintainability, and adaptability of the data processes and workflows.

Looks good, huh? …


Opa! Voltei! Bem rápido, mas voltei!

Estou realmente sem tempo pra postar algo, mas tem algo bem interessante que aparentemente não é de conhecimento de muitas pessoas quando falamos de Terraform e IAM (Identity Access Management), que são os modos additives e authoritatives.

Vamos pegar por exemplo o conceito a nível de projetos, mas pode ser utilizado em nível de organização ou folder também.

O tipo de recurso é o"google_project_iam" no Terraform. Vamos falar nos 2 principais tipos de gerenciamento:


Esses dias eu estava fazendo um workshop sobre conceitos de Google Cloud Computing para um cliente, e quando chegamos no BigQuery (DW do GCP + outras coisas), uma das perguntas foi: “eu tenho uma tabela em que algumas colunas não podem ser visíveis para todos. Tenho como limitar isso para alguns grupos sem ter que me preocupar em criar views, ou outra tabela?”

Minha resposta foi: “já li alguma coisa a respeito, mas volto depois com a resposta mais concreta.”. Fui pesquisar e cai nisso aqui:

https://cloud.google.com/bigquery/docs/column-level-security-intro

Retornei para o cliente com essa documentação, mas falei: “vou fazer uns testes…


Hey you!!! Happy new year!! 2020 was a strange year, but I can’t complain. Lot of things, knowledge, worked a lot. So, let’s keep going in 2021!!!

So, let’s talk a little about security. I’ve been working within an interesting project to make GCP Security Command Center to work with some external tools and automation with Cloud Functions. And guess what: as I love automation, here’s what I’ve being studying:


Hey you!!! Happy new year!! 2020 was a strange year, but I can’t complain. Lot of things, knowledge, worked a lot. So, let’s keep going in 2021!!!

So, let’s talk a little about security. I’ve been working within an interesting project to make GCP Security Command Center to work with some external tools and automation with Cloud Functions. And guess what: as I love automation, here’s what I’ve being studying:


I’ve finally got some “free” time to learn a little more about Machine Learning. So, that said, I’m almost crazy with Math (again, same feeling of 90's).

So, I'm using this to study:

It has been really productive, even for me (I'm not a developer, so it's hard sometimes to go ahead with a few topics). So, my starting point was to run the Titanic study case from Kaggle but using now Bigquery…


Data Pipeline is basically move some data from one place to another. This move may have or not changes, processing, …

For companies that works with big ammount of data, this is basic job. Data is received, transformed, enriched with other data if needed, moved to data lakes or any other place, and most of times, finished with some beautiful dashboard.

We have in Google Cloud the Data Fusion service. Basically is a service for efficiently building ETL/ELT data pipelines. It uses Cloud Dataproc cluster to perform all transforms in the pipeline. …


If you don’ know the best way to use it, of course!

BigQuery it’s the Data Warehouse solution fom Google Cloud. And it’s REALLY amazing what you can do with this tool!!! But at same time, it can take you to ruin!! Some of great points about BigQuery is that is serverless, highly-scalable and you can plugin a lot of sources to analyze.

There’s also a lot of main points like multi-cloud analytics, machine learning with SQL, GIS, BI, Sheets connections, … sky is the limit to analysis.

The only point here is about the pricing. BigQuery charges for data…


Terraform is great. It helps you keep things clear and allow you to provision almost everything in the Cloud (I know, not only in Cloud, but this is what I've being using).

Photo by Sai Kiran Anagani on Unsplash

One more challenge that I took to myself when I got "in charge" for Terraform in a new project… 1 month to certificate (less, actually). As I did about AWS on this post, again, I plan my "journey" to take the HashiCorp Certified: Terraform Associate Certification.

Marcelo Marques

enjoy the journey. do the right thing.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store