Prerequisites

Before you follow this guide, make sure you have:

1. Google Cloud SDK (gcloud) installed
   Install the official Google Cloud SDK for your operating system:

   • Docs: https://cloud.google.com/sdk/docs/install

2. Two separate Google accounts

   • One personal Google account (e.g., your Gmail)

   • One office/work Google account managed by your organization

3. Existing gcloud configurations (optional but recommended)
   It’s helpful to have named gcloud configurations for each account, for example:

   • personal

   • office (or any name you prefer for your office account)

   You can check existing configurations with:

   gcloud config configurations list
   

4. Basic familiarity with your shell
   You should be comfortable with:

   • Editing your shell config file (~/.bashrc or ~/.zshrc)

   • Running commands in your terminal

With these prerequisites in place, you’re ready to set up separate ADC profiles and account-switching aliases.

• Create separate Application Default Credentials (ADC) profiles for personal and office accounts

• Switch between them quickly using simple shell aliases

Create an ADC profile for your personal account

First, log in with your personal Google account using gcloud:

gcloud auth application-default login

This command will print a URL. Open it in your browser and log in using your personal Gmail account. Once you finish authentication, gcloud will generate an ADC file at:

~/.config/gcloud/application_default_credentials.json

Create adc_personal.json

Now copy the newly generated ADC file to a dedicated file for your personal account:

cp ~/.config/gcloud/application_default_credentials.json 
~/.config/gcloud/adc_personal.json

We create a separate file because a single application_default_credentials.json cannot store multiple accounts. Each account needs its own ADC file.

Create an ADC profile for your office account

Repeat the process, this time using your office Google account.

First, log in with your office email:

gcloud auth application-default login

Again, open the URL in your browser and authenticate with your office account. After login, the same application_default_credentials.json file will be updated with your office credentials.

Create adc_office.json

Now copy that file to a dedicated ADC file for your office account:

cp ~/.config/gcloud/application_default_credentials.json 
~/.config/gcloud/adc_office.json

At this point, you should have:
~/.config/gcloud/adc_personal.json → personal credentials ~/.config/gcloud/adc_office.json → office credentials

Create GCP account switcher aliases

To quickly switch between accounts, you can define shell aliases that:

1. Activate the correct gcloud configuration

2. Set the default project

3. Point GOOGLE_APPLICATION_CREDENTIALS to the right ADC file

First Check which shell you’re using Run:

echo $0

If the output contains bash, edit:

vi ~/.bashrc

If it contains zsh, edit:

vi ~/.zshrc

Add the aliases

Append the following lines at the end of your shell config file (.bashrc or .zshrc). Make sure to replace <project_id> with your actual GCP project IDs.

###-------- GCP ACCOUNT SWITCHER -------------
###--- PERSONAL SWITCH ---

alias gcp-personal=' gcloud config configurations activate personal; 
gcloud config set project <personal_project_id>; 
export GOOGLE_CLOUD_PROJECT="<personal_project_id>"; 
export GOOGLE_APPLICATION_CREDENTIALS="$HOME/.config/gcloud/adc_personal.json"; 
echo "Switched to PERSONAL (Gmail)"'

###--- OFFICE SWITCH ---

alias gcp-office=' gcloud config configurations activate office; 
gcloud config set project <office_project_id>; 
export GOOGLE_CLOUD_PROJECT="<office_project_id>"; 
export GOOGLE_APPLICATION_CREDENTIALS="$HOME/.config/gcloud/adc_office.json"; 
echo "Switched to Office (Office)"' 

After editing the file, reload your shell configuration:

# For bash
source ~/.bashrc

# For zsh
source ~/.zshrc

Switching between accounts

Now you can switch accounts with a single command:

##Switch to personal GCP account
gcp-personal

##Switch to office GCP account
gcp-office

Each alias will:

• Activate the right gcloud configuration

• Set the appropriate default project

• Point GOOGLE_APPLICATION_CREDENTIALS to the correct ADC file

Conclusion

This setup helps you avoid accidental deployments to the wrong project and makes working with multiple GCP accounts much smoother.

What we’ll cover

• Cloud Storage FUSE

• Service Account

• Linux User

• GCP VM

• Cloud KMS

What you’ll need

• Google Cloud Admin access

• Access to VM and a Bucket

• Cloud KMS access

Here is flowchart on this lab

Lets first create the Cloud KMS keyring and Cryptographic Key

Setup Cloud KMS

Cloud KMS help in building the data sovereignty between GCS bucekt and VM, It ensures that even if someone get access to physical disk in Google data’s center, they cannot read your files without your cryptographic keys.

We are using this Cloud KMS to setup most secure infra for our data, from virtual to physical data will be secure in this data transaction

gcloud kms keyrings create secure-bucket-keyring --location us-central1

This keyring will be use by the Cloud bucket and VM to decrypt the data

Create Service account to user to use this KMS key

export SA_NAME="secure-vm-sa"
gcloud iam service-accounts create $SA_NAME --display-name "Secure VM Access"

Grant permission to us Cloud KMS key

export REGION="us-central1"
export KEY_RING="secure-bucket-keyring"
export KEY_NAME="secure-vm-bucket-key"
export SA_EMAIL="\(SA_NAME@\)PROJECT_ID.iam.gserviceaccount.com"
gcloud kms keys add-iam-policy-binding $KEY_NAME \
    --location $REGION \
    --keyring $KEY_RING \
    --member "serviceAccount:$SA_EMAIL" \
    --role "roles/cloudkms.cryptoKeyEncrypterDecrypter"

Now will grant the bucket to access the Cloud KMS key

export PROJECT_ID=$(gcloud config get-value project)
export PROJECT_NUMBER=\((gcloud projects describe \)PROJECT_ID --format="value(projectNumber)")
export STORAGE_AGENT="service-$PROJECT_NUMBER@gs-project-accounts.iam.gserviceaccount.com"
gcloud kms keys add-iam-policy-binding $KEY_NAME \
    --location $REGION \
    --keyring $KEY_RING \
    --member "serviceAccount:$STORAGE_AGENT" \
    --role "roles/cloudkms.cryptoKeyEncrypterDecrypter" 

Now we will create Bucket

Bucket setup

We will apply four security layers in a single command:

1. Enforce Public Access Prevention: Hard block on "allUsers" (internet).

2. Uniform Bucket-Level Access: Disables messy ACLs (Access Control Lists); forces strict IAM usage.

3. Default CMEK Encryption: Forces your KMS key on every file immediately.

4. Versioning: Protects against ransomware or accidental deletion (you can "undelete" files).

BUCKET_NAME="secure-bucket-lab-2025"
REGION="us-central1"
PROJECT_ID=$(gcloud config get-value project)
KMS_KEY="projects/\(PROJECT_ID/locations/\)REGION/keyRings/secure-bucket-keyring/cryptoKeys/secure-vm-bucket-key"

Note: Create a bucket with unique name please don’t use this name

Now let create secure bucket with Cloud KMS key, remove public access, and add uniform bucket level access and enable versioning

gcloud storage buckets create gs://$BUCKET_NAME \
    --location=$REGION \
    --project=$PROJECT_ID \
    --public-access-prevention \
    --uniform-bucket-level-access \
    --default-encryption-key=$KMS_KEY

Enable versioning to bucket

gcloud storage buckets update gs://$BUCKET_NAME --versioning

Grant a service account access to bucket

gcloud storage buckets add-iam-policy-binding gs://$BUCKET_NAME \
    --member "serviceAccount:$SA_EMAIL" \
    --role "roles/storage.objectUser"

Now we will create the VM

VM_NAME="secure-vm"
ZONE="us-central1-a"
PROJECT_ID=$(gcloud config get-value project)
SA_EMAIL="secure-vm-sa@$PROJECT_ID.iam.gserviceaccount.com"

gcloud compute instances create $VM_NAME \
    --zone=$ZONE \
    --machine-type=e2-medium \
    --image-family=ubuntu-2204-lts \
    --image-project=ubuntu-os-cloud \
    --service-account=$SA_EMAIL \
    --scopes=cloud-platform \
    --shielded-secure-boot \
    --shielded-vtpm \
    --shielded-integrity-monitoring \
    --tags=secure-ssh

We added these thing to make it more secure

• --shielded-secure-boot: Ensures the OS hasn't been tampered with before it loads.

• --service-account: It never uses the "Default" editor account, only your restricted one.

• --tags=secure-ssh: We will use this tag to create a specific firewall rule next.

Now secure the Firewall to allow access with only Cloud IAP

gcloud compute firewall-rules create allow-ssh-ingress-from-iap \
    --direction=INGRESS \
    --action=allow \
    --rules=tcp:22 \
    --source-ranges=35.235.240.0/20 \
    --target-tags=secure-ssh

Use this ssh command to access the VM in Cloud IAM ssh tunnel

gcloud compute ssh \(VM_NAME --zone=\)ZONE --tunnel-through-iap

Install Dependencies on VM

Google Cloud Storage FUSE

export GCSFUSE_REPO=gcsfuse-`lsb_release -c -s`
echo "deb https://packages.cloud.google.com/apt $GCSFUSE_REPO main" | sudo tee /etc/apt/sources.list.d/gcsfuse.list
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -

sudo apt-get update
sudo apt-get install -y gcsfuse

Check your user id

id <username>

Create new users and group then

sudo groupadd -g 2000 app_group
sudo useradd -u 2000 -g 2000 -m -s /bin/bash app_user

create secure folder and give access to the particular user

sudo mkdir -p /home/app_user/secure_data
sudo chown app_user:app_group /home/app_user/secure_data
sudo chmod 700 /home/app_user/secure_data

User mount to fuse.conf file

sudo sed -i 's/#user_allow_other/user_allow_other/g' /etc/fuse.conf

Mount the bucket data to persistent volume

BUCKET_NAME="secure-bucket-lab-2025" 
echo "$BUCKET_NAME /home/app_user/secure_data gcsfuse rw,_netdev,allow_other,implicit_dirs,uid=2000,gid=2000,file_mode=600,dir_mode=700,noexec,nosuid,nodev 0 0" | sudo tee -a /etc/fstab
sudo mount -a

Now switch to user and verify the folder

sudo su - app_user
ls -ld secure_data/

This will the folder secure_data with drwx------ 1 app_user app_group 0

Now verify the mount volume

df -h /home/app_user/secure_data

Here you can see the avail size 1.0P which is significant saying Google Storage bucket in petabyte of storage

Test data access

Push object into bucket

Now ls inside the secure_data folder

In this lab, we explore securing data access between Google Cloud Storage (GCS) buckets and Google Cloud Platform (GCP) virtual machines (VMs) using Cloud Key Management Service (KMS) and other security measures. Key steps include setting up a KMS keyring and cryptographic key, creating a service account for secure VM access, configuring a secure bucket with four layers of security (preventing public access, enforcing uniform bucket-level access, default CMEK encryption, and versioning), and setting up the VM with specific security configurations. We also cover using Cloud Storage FUSE for data access, creating secure user groups, and mounting the bucket data as a persistent volume. Finally, we verify data accessibility and security configurations on the VM.

Step 1: Organize Your Folders

To maintain multiple GitHub account in your system best way to do is create separate for each of them such as

mkdir -p ~/Personal
mkdir -p ~/Work
mkdir -p ~/client1

Here in each of folder we can create work in each folder and each folder can align to correct git credentials

Step 2: Create Config Files for Each Account

nano ~/.gitconfig-personal

Add the following content. Replace the email with your actual personal email.

[user]
    name = your-personal-github-username
    email = your-personal-email@example.com

nano ~/.gitconfig-work

Add the following content. Replace the email with your actual work email.

[user]
    name = your-work-github-username
    email = your-work-email@example.com

nano ~/.gitconfig-client-1

Add the following content. Replace the email with your actual client1 email.

[user]
    name = your-client1-github-username
    email = your-client1-email@example.com

Step 3: Update Your Main Git Config

Now, edit your main ~/.gitconfig file to automatically use the correct config file based on the project's path.

Open ~/.gitconfig and make it look like this. This sets your personal account as the default and creates rules for your work directory.

Usually when your are working on work laptop or system you should keep default as work username and email then Set your work account as the default on your work laptop to ensure commits use the correct credentials

# This is your default (work) user config
[user]
    name = your-work-github-username
    email = your-personal-email@example.com

# --- Automatic Profile Switching ---
# If the project is inside ~/dev/work/, use the work config
[includeIf "gitdir:~/Personal"]
    path = ~/.gitconfig-personal

[includeIf "gitdir:~/client1"]
    path = ~/.gitconfig-client1

With this setup, Git will automatically use the correct name and email for your commits.

Step 4: Manage SSH Keys for Authentication

To push and pull from different accounts on the same service (like GitHub), you need separate SSH keys.

1. Generate Two SSH Keys

Create a unique SSH key for each account.

# Personal key
ssh-keygen -t ed25519 -C "your-personal-email@example.com" -f ~/.ssh/id_ed25519_personal

# Work key
ssh-keygen -t ed25519 -C "your-work-email@example.com" -f ~/.ssh/id_ed25519_work

# Client key
ssh-keygen -t ed25519 -C "your-client1-email@example.com" -f ~/.ssh/id_ed25519_client1

If you’re thinking what that id_ed25519: This is the cryptographic algorithm used to generate the key. Ed25519 is a modern, secure, and fast choice for SSH keys.

When prompted for a passphrase, you can either enter one for extra security or press Enter to skip.

2. Add Keys to Your Accounts

Copy the contents of each public key (.pub file) and add them to your corresponding GitHub accounts in Settings > SSH and GPG keys.

Copy the contents of each public key (.pub file) and add them to your corresponding GitHub accounts in Settings > SSH and GPG keys.

# Copy your personal public key
pbcopy < ~/.ssh/id_ed25519_personal.pub

# Copy your work public key
pbcopy < ~/.ssh/id_ed25519_work.pub

# Copy your client public key
pbcopy < ~/.ssh/id_ed25519_client1.pub

3. Configure SSH to Use the Correct Key

Create or edit your SSH config file at ~/.ssh/config. This file tells your computer which key to use for which account.

nano ~/.ssh/config

Add the following configuration:

# Personal GitHub account
Host github.com-personal
    HostName github.com
    User git
    IdentityFile ~/.ssh/id_ed25519_personal
    IdentitiesOnly yes

# Work GitHub account
Host github.com-work
    HostName github.com
    User git
    IdentityFile ~/.ssh/id_ed25519_work
    IdentitiesOnly yes

# Client1 GitHub account
Host github.com-client1
    HostName github.com
    User git
    IdentityFile ~/.ssh/id_ed25519_client1
    IdentitiesOnly yes

4. Clone Repos Using Custom Hosts

From now on, when you clone a repository, you must use the custom host you just defined. This ensures the correct SSH key is used.

• To clone a personal repo:

    git clone git@github.com-personal:your-personal-username/personal-repo.git
  

• To clone a work repo:

    git clone git@github.com-work:your-work-username/work-repo.git
  

• To clone a client1 repo:

    git clone git@github.com-work:your-client1-username/client1-repo.gitYour
  

system is now fully configured to manage both accounts automatically.

5. Update Your Git Repository's Remote URL

Finally, navigate to the local repository you want to fix and tell it to use the new SSH URL.

• For a personal repository (like your personal project):

    cd ~/Personal/vazid
    git remote set-url origin git@github.com-personal:your-personal-usernmae/repo.git
  

• For a work repository:

    cd ~/Work/my-work-project
    git remote set-url origin git@github.com-work:your-work-username/work-repo.git
  

• For a work repository:

    cd ~/client1/my-work-project
    git remote set-url origin git@github.com-client1:your-client1-username/client1-repo.git
  

You are now fully set up to push and pull using the correct account automatically.

Thank You

Key Changes of the Coffee House Theme

1. Coffee-Inspired Palette

The theme features rich browns, warm golds, and muted jewel tones that evoke the coziness of a coffee shop:

• Foreground: Creamy coffee #E5D3B3

• Background: Dark espresso #1C1715

• Cursor: Golden caramel #D4A373

2. Transparency

The background is slightly transparent (background-opacity = 0.85), blending seamlessly with your desktop for a modern and sophisticated look. and change as per your aesthetics

3. Title Customization

The terminal title, set to "☕ Coffee House", adds a unique and personal touch.

4. Font Settings

With the clean and modern Source Code Pro font, the theme ensures excellent readability.

The configuration file for Ghostty terminal is typically located at:

~/.config/ghostty/config

Update the config file to the below config file

# Ghostty Terminal Configuration
# Coffee House

# Window titlebar colors (only effective with ghostty theme)
window-titlebar-background = #362F2D  
window-titlebar-foreground = #E5D3B3  

# Terminal transparency
background-opacity = 0.85

# Title for the terminal
title  = "☕ Coffee House"

# Window theme
# Remove this if you are not using linux
window-theme = ghostty

# Coffee-inspired color scheme with chocolate accents
foreground = #E5D3B3  
background = #1C1715
cursor-color = #D4A373 

# Normal colors (coffee and chocolate tones)
palette = 0=#3A2C28  
palette = 1=#A64D4D  
palette = 2=#7B6D49  
palette = 3=#C9A16C  
palette = 4=#6E4F4B  
palette = 5=#A67D94  
palette = 6=#8D7B6E  
palette = 7=#E5D3B3  

# Bright colors (wrapper and highlights)
palette = 8=#4D3E39  
palette = 9=#BF7F7F  
palette = 10=#9E8D6C 
palette = 11=#D4B98B  
palette = 12=#957572  
palette = 13=#BA94B4  
palette = 14=#B7A493  
palette = 15=#FFF5E1  

# Font settings
font-family = "Source Code Pro"
font-size = 12
font-style = "Medium"

After saving the changes, restart Ghostty terminal to apply the new theme.

or use shortcut key shift+ctrl+,

Here are some Screenshot for the updated ghostty terminal

Share Your Setup!

Enjoy coding in your cozy new terminal setup. Happy brewing! ☕

• Perform system updates with a single command.

• Schedule updates using cron jobs for convenience.

• Log every update for better tracking and troubleshooting.

Prerequisites

Before using this script, ensure you have:

• Root/sudo access on your Linux system.

• Basic knowledge of Linux commands.

• Sufficient disk space for updates.

• A stable internet connection.

The Script

#!/bin/bash

RELEASE_FILE="/etc/os-release"
LOG_DIR="/var/log/system_updates"
DATE=$(date '+%Y-%m-%d_%H-%M-%S')
LOG_FILE="$LOG_DIR/system_update_$DATE.log"
LOCK_FILE="/tmp/system_update.lock"
MAX_RETRIES=3
RETRY_DELAY=5

cleanup() {
    rm -f "$LOCK_FILE"
    log_message "------------------Script execution completed"
}

trap cleanup EXIT

log_message() {
    local message="$1"
    local timestamp=$(date '+%Y-%m-%d %H:%M:%S')
    echo "$timestamp - $message" | sudo tee -a "$LOG_FILE"
}

check_system_resources() {
    local min_space=1000000
    local available_space=$(df /usr -k | awk 'NR==2 {print $4}')

    if [ "$available_space" -lt "$min_space" ]; then
        log_message "!!!!! ERROR: Insufficient disk space. Required: 1GB, Available: $((available_space / 1024))MB !!!!!"
        exit 1
    fi
}

create_lock() {
    if [ -f "$LOCK_FILE" ]; then
        local pid=$(cat "$LOCK_FILE")
        if kill -0 "$pid" 2>/dev/null; then
            log_message "*****Another update process is running. Exiting"
            exit 1
        fi
    fi
    echo $$ > "$LOCK_FILE"
}

create_log_directory() {
    if [ ! -d "$LOG_DIR" ]; then
        log_message "Creating log directory..."
        if sudo mkdir -p "$LOG_DIR"; then
            sudo chmod 750 "$LOG_DIR"
            log_message "Log directory created at $LOG_DIR"
        else
            log_message "!!!! ERROR: Failed to create log directory !!!!"
            exit 1
        fi
    fi
}

check_internet() {
    local retry=0
    while [ $retry -lt $MAX_RETRIES ]; do
        if ping -c 1 8.8.8.8 >/dev/null 2>&1; then
            return 0
        fi
        retry=$((retry + 1))
        log_message "Waiting for internet connection (attempt $retry/$MAX_RETRIES)"
        sleep $RETRY_DELAY
    done
    log_message "!!!!! ERROR: No internet connection !!!!!"
    return 1
}

update_arch() {
    log_message "----------Updating Arch Linux------------"

    if ! sudo pacman -Sy --noconfirm 2>&1 | sudo tee -a "$LOG_FILE"; then
        log_message "ERROR: Failed to sync package databases"
        return 1
    fi

    if sudo pacman -Su --noconfirm 2>&1 | sudo tee -a "$LOG_FILE"; then
        log_message "--------------Arch Linux system updated successfully"
        sudo paccache -r
    else
        log_message "!!!!! ERROR: Failed to update Arch Linux system !!!!!"
        return 1
    fi
}

update_apt() {
    log_message "--------------Updating APT-based system-------------"

    export DEBIAN_FRONTEND=noninteractive

    local retry=0
    while [ $retry -lt $MAX_RETRIES ]; do
        if sudo apt-get update -y 2>&1 | sudo tee -a "$LOG_FILE" && \
           sudo apt-get upgrade -y 2>&1 | sudo tee -a "$LOG_FILE" && \
           sudo apt-get dist-upgrade -y 2>&1 | sudo tee -a "$LOG_FILE"; then
            log_message "---------------APT system updated successfully"
            sudo apt-get autoremove -y
            sudo apt-get clean
            return 0
        fi
        retry=$((retry + 1))
        log_message "Update attempt $retry failed. Retrying in $RETRY_DELAY seconds..."
        sleep $RETRY_DELAY
    done
    log_message "!!!!! ERROR: Failed to update APT system after $MAX_RETRIES attempts !!!!!"
    return 1
}

update_redhat() {
    log_message "--------------Updating RedHat-based system---------------"

    local retry=0
    while [ $retry -lt $MAX_RETRIES ]; do
        if sudo dnf update -y 2>&1 | sudo tee -a "$LOG_FILE" && \
           sudo dnf upgrade -y 2>&1 | sudo tee -a "$LOG_FILE"; then
            log_message "---------------RedHat system updated successfully"
            sudo dnf clean all
            return 0
        fi
        retry=$((retry + 1))
        log_message "Update attempt $retry failed. Retrying in $RETRY_DELAY seconds..."
        sleep $RETRY_DELAY
    done
    log_message "!!!! ERROR: Failed to update RedHat system after $MAX_RETRIES attempts !!!!"
    return 1
}

main() {
    if [ "$EUID" -ne 0 ]; then
        echo "This script must be run as root. Please use sudo." >&2
        exit 1
    fi

    create_lock
    create_log_directory

    log_message "------------------------Starting system update"

    check_system_resources

    if ! check_internet; then
        exit 1
    fi

    if grep -qi "arch" "$RELEASE_FILE"; then
        update_arch
    elif [ -d /etc/apt ]; then
        update_apt
    elif grep -qiE "redhat|fedora|centos" "$RELEASE_FILE"; then
        update_redhat
    else
        log_message "!!!! ERROR: Unsupported or unrecognized Linux distribution !!!!!"
        exit 1
    fi

    log_message "--------------------------System update process completed"
}

main

Setting Up the Script

1. Download the script.

2. Make it executable: chmod +x update.sh

3. Move it to a system directory: sudo mv update.sh /usr/local/bin/update

4. Set proper permissions: sudo chmod 755 /usr/local/bin/update

Automating the Script with Cron

To schedule daily updates, create a cron job:

1. Open the cron editor:

    sudo crontab -e
   

2. Add the following line:

    0 0 * * * /usr/local/bin/update
   

This schedules the script to run every day at midnight.

Managing Log Files

To prevent log file accumulation, create another cron job to delete logs older than 30 days:

0 0 * * * find /var/log/system_updates/ -type f -mtime +30 -exec rm {} \;

Conclusion

By automating your system updates with this script, you can save time and ensure your Linux system is secure and up-to-date. It’s especially useful for servers and critical systems.

Give this script a try, and let me know how it works for you! Got questions or suggestions? Feel free to reach out via email.

Don’t forget to share this with your fellow Linux enthusiasts!

Both authorized views and materialized views are powerful tools in BigQuery that offer improved data access and performance. However, they have key differences in their functionalities and applications:

Authorized Views:

• Definition: Automatically created views based on queries with a WHERE clause.

• Functionality: Pre-filter data based on the WHERE clause, improving query performance by reducing scanned data.

• Read-only: Cannot be directly modified.

• Benefits:

  • Efficient query performance: Reduces scanned data, leading to faster queries.

  • Simplified data access: Simplifies complex queries with pre-filtered data.

  • Data access control: Inherits permissions from the underlying table or view, offering easier control.

To create an Authorized View

Steps to set up Authorized View in BigQuery with Principle of least Privilege | PDE & PCA Concepts

Materialized Views:

• Definition: User-created views that store pre-computed results of a query.

• Functionality: Materialized views physically store data based on the query, significantly speeding up repeated queries with the same logic.

• Updatable: This can be automatically or manually updated based on changes in the underlying data.

• Benefits:

  • Extreme query performance: Provide the fastest query response for frequently used queries.

  • Reduced processing overhead: Eliminates repeated calculations for frequently used queries.

  • Scalability: Enables efficient handling of large datasets for repeated queries.

• Example: SQL

    CREATE MATERIALIZED VIEW [view_name] AS
    SELECT [column1], [column2], SUM([column3]) AS total_value
    FROM [dataset_name.][table_name]
    GROUP BY [column1], [column2];
  

  This query creates a materialized view named [view_name] that stores pre-calculated total values for specific columns and groups.

Choosing the Right Option:

The best choice between authorized views and materialized views depends on your specific needs:

• Frequently used queries: Materialized views are ideal for queries used repeatedly, significantly improving performance.

• Large datasets: Materialized views offer faster processing for large datasets involved in repeated queries.

• Data complexity: Materialized views are suitable for complex queries with aggregations and calculations.

• Data update frequency: Authorized views are better if the underlying data changes frequently, ensuring the view reflects the latest information.

• Data access control: Authorized views inherit permissions from the base data, simplifying control.

How to optimize table performance in Bigquery

Optimizing table performance in Google BigQuery involves various strategies, including optimizing table structure, using efficient queries, and leveraging features provided by BigQuery. Here are some tips to enhance table performance:

1. Partition and Cluster Tables:

• Partitioning: Divide large tables into smaller, more manageable partitions based on a date or timestamp column. This can significantly reduce the amount of data scanned during queries.

• Clustering: Use clustering to organize data within partitions based on one or more columns. Clustering improves query performance by minimizing the amount of data that needs to be read.

Anjan GCP Data Engineering - YouTube has explained both cluster and partition tables

Big Query Table Partitions with Examples

Big Query Clustered Tables with Examples

2. Use Appropriate Data Types:

Choose the most suitable data types for your columns to minimize storage and improve query performance. Avoid unnecessary use of STRING for numeric or boolean data.

3. Optimize Schema Design:

Normalize or denormalize your schema based on query patterns. Denormalization can reduce the need for JOIN operations and improve query performance.

4. \Avoid SELECT :*

Instead of selecting all columns using SELECT *, explicitly list only the columns you need. This reduces the amount of data read and improves query speed.

5. Control Query Output Size:

Use LIMIT and WHERE clauses to control the amount of data retrieved. Minimize the data scanned by only retrieving the necessary rows.

6. Use Batch Loading for Large Data:

When loading large amounts of data, consider using batch loading. Load data in larger chunks rather than row by row for better performance.

7. Optimize JOIN Operations:

If JOIN operations are necessary, use appropriate JOIN types and conditions. Ensure that the joined columns are properly indexed.

8. Optimize Query Syntax:

Write efficient queries. Review query execution plans using the EXPLAIN statement to identify opportunities for optimization.

9. Materialized Views (Limited):

Consider using views or materialized views to precompute and store aggregated data, but be aware that as of my last knowledge update, BigQuery doesn't have native materialized views. You may need to manually update tables or views periodically.

11. Monitor and Analyze Performance:

Use BigQuery's monitoring tools to analyze query performance. Review the Query Execution Details page in the Cloud Console for insights into query performance.

What is the slot in BigQuery

Slot = resource which is automatically provided by BigQuery

BigQuer Slot

How to query data from the Amazon s3 in BigQuery using BigQuery omni

BigQuery Omni allows querying data directly from external sources like Amazon S3 without needing to load it into BigQuery first. This offers several advantages, including:

• Real-time access: Analyze data as soon as it's available in S3 without waiting for loading.

• Cost-efficiency: Avoid unnecessary storage costs by querying data directly from the source.

• Scalability: Process massive datasets without worrying about BigQuery storage limitations.

Here's how to query data from Amazon S3 to BigQuery Omni:

1. Set up BigQuery Omni:

• Enable BigQuery Omni in your Google Cloud project.

• Create a connection to your Amazon S3 account in BigQuery.

• Specify IAM permissions for BigQuery to access your S3 data.

2. Create a BigLake table:

• Define a BigLake table in BigQuery that references the data location in your S3 bucket.

• Specify the data schema and format (e.g., CSV, JSON, Avro).

• Optionally, configure partitioning and clustering for optimal query performance.

3. Write your SQL query:

• Use standard BigQuery SQL syntax to query the data in the BigLake table.

• BigQuery automatically fetches the data from S3 on-demand as needed for the query.

• You can perform joins, aggregations, and other operations on the S3 data as if it were stored in BigQuery.

4. Export results:

• Export the results to other destinations like Google Cloud Storage or BigQuery tables.

Here are some points to consider when querying data from Amazon S3 to BigQuery Omni:

• Data access control: Ensure you have proper IAM permissions granted to BigQuery for accessing your S3 data.

• Data format: BigQuery supports various data formats, but ensure the format used in S3 is compatible with your BigLake table definition.

• Data size: Large datasets may require additional configuration for efficient querying and performance.

• Query complexity: Complex queries may require more resources and potentially incur network costs for data retrieval from S3.

What are the different processes to export data from the BigQuery

BQ command to load the data from GCS to BigQuery

bq load --source_format [FORMAT] [DATASET.TABLE] [GCS_URI]

What operators used Airflow DAG to load data from GCS to BigQuery

Airflow offers several operators for loading data from Google Cloud Storage (GCS) to BigQuery. Here are the most commonly used ones:

GCStoBigQueryOperator:

This operator is the most common choice for loading data from GCS to BigQuery.

• Features:

  • Supports various data formats like CSV, JSON, Avro, and Parquet.

  • Offers options to specify schema, compression, and write disposition.

  • Allows triggering load jobs based on file arrival or time intervals.

• Example:

Python

from airflow import DAG
from airflow.providers.google.cloud.operators.bigquery import GCSToBigQueryOperator

with DAG(
    dag_id="gcs_to_bigquery",
    start_date=datetime(2023, 12, 13),
    schedule_interval="@daily",
) as dag:

    load_data = GCSToBigQueryOperator(
        task_id="load_data",
        bucket="my_bucket",
        source_object="data.csv",
        dataset="my_dataset",
        table="my_table",
    )

Difference in DELETE table and Truncate table in BigQuery

TRUNCATE and DELETE are both SQL commands used to remove data from a table, but they differ in their functionality and the way they accomplish the task.

1. DELETE:

   • The DELETE statement is used to remove rows from a table based on a specified condition or without any condition (which would delete all rows).

   • It is a logged operation, meaning that each deleted row is recorded in the transaction log, allowing for the possibility of rolling back the transaction.

   • The DELETE statement is more flexible and can be used with a WHERE clause to specify a condition for deleting rows.

Example:

    DELETE FROM table_name WHERE condition;

2. TRUNCATE:

   • The TRUNCATE statement is used to remove all rows from a table quickly and efficiently.

   • Unlike DELETE, TRUNCATE is not logged for each individual row; instead, it deallocates the data pages used by the table.

   • TRUNCATE is a faster operation than DELETE because it doesn't generate individual row deletion statements, and it doesn't log individual row deletions.

Example:

    TRUNCATE TABLE table_name;

Key Differences:

• DELETE is a row-level operation, allowing you to delete specific rows based on a condition, whereas TRUNCATE is a table-level operation that removes all rows.

• DELETE is slower than TRUNCATE because it logs each row deletion, making it suitable for smaller-scale row deletions where logging is necessary.

• TRUNCATE is more efficient for removing all rows from a table, but it cannot be used when the table is referenced by a foreign key constraint or if it participates in an indexed view.

In summary, use DELETE when you need to selectively remove specific rows or when logging individual row deletions is necessary. Use TRUNCATE when you want to quickly remove all rows from a table

SQL

One of the main parts of the BigQuery interview

Question:

Write a query to get the below output

Tables

Employee table:
+----+-------+--------+--------------+
| id | name  | salary | departmentId |
+----+-------+--------+--------------+
| 1  | Joe   | 70000  | 1            |
| 2  | Jim   | 90000  | 1            |
| 3  | Henry | 80000  | 2            |
| 4  | Sam   | 60000  | 2            |
| 5  | Max   | 90000  | 1            |
+----+-------+--------+--------------+
Department table:
+----+-------+
| id | name  |
+----+-------+
| 1  | IT    |
| 2  | Sales |
+----+-------+

Output

+------------+----------+--------+
| Department | Employee | Salary |
+------------+----------+--------+
| IT         | Jim      | 90000  |
| Sales      | Henry    | 80000  |
| IT         | Max      | 90000  |
+------------+----------+--------+

Solution:

SELECT d.name as Department,
       e.name as Employee,
       e.salary as Salary
FROM employee e
JOIN department d ON e.departmentId = d.id
WHERE e.salary >= (SELECT MAX(salary) FROM employee)
ORDER BY e.salary DESC;

Question:

Find the average salary for each department. Display the department name and the average salary.

Solution:

SELECT d.name AS Department,
       AVG(e.salary) AS AverageSalary
FROM employee e
JOIN department d ON e.departmentId = d.id
GROUP BY d.name;

Question:

List the employees who have a salary greater than the average salary across all departments. Display the employee name, salary, and department name.

WITH AvgSalaryCTE AS (
  SELECT AVG(salary) AS AvgSalary
  FROM employee
)

SELECT e.name AS Employee,
       e.salary AS Salary,
       d.name AS Department
FROM employee e
JOIN department d ON e.departmentId = d.id
CROSS JOIN AvgSalaryCTE
WHERE e.salary > AvgSalary;

Question:

Find the department with the highest total salary. Display the department name and the total salary.

SELECT d.name AS Department,
       SUM(e.salary) AS TotalSalary
FROM employee e
JOIN department d ON e.departmentId = d.id
GROUP BY d.name
ORDER BY TotalSalary DESC
LIMIT 1;

Question:

List the employees and their salaries in descending order of salary. For employees with the same salary, order them alphabetically by name.

Solution:

sqlCopy codeSELECT name AS Employee,
       salary AS Salary
FROM employee
ORDER BY salary DESC, name;

Question:

Find the department(s) where the average salary is greater than a specified value (e.g., 80000). Display the department name and the average salary.

Solution:

sqlCopy codeSELECT d.name AS Department,
       AVG(e.salary) AS AverageSalary
FROM employee e
JOIN department d ON e.departmentId = d.id
GROUP BY d.name
HAVING AVG(e.salary) > 80000;

Question:

What is the output of Table A and Table B from Inner Join, Left Outer Join and Full Join?

Table A:

1
2
2
3
4
6
NULL
NULL

Table B:

1
1
2
7
8
null

Solution:

Inner Join

Result:

1
2
2
3
4
6
NULL

Left Outer Join

Result:

1
2
2
3
4
6
NULL
NULL

Full Join

Result:

1
2
3
4
6
7
8
NULL
NULL

Question:

Select Concat('a',null,'b'):

Result:

null

In SQL, concatenating anything with NULL results in NULL. If you want to handle NULL values differently, you can use the COALESCE function or the CONCAT function with multiple arguments.

SELECT CONCAT('a', COALESCE(null, ''), 'b');
-- or
SELECT CONCAT('a', nullif('', null), 'b');

Both of these queries will result in 'ab'.

Data reconciliation is an important part of the data migration. After the completion of data migration we have to compare data from the source data to destination data and here we are going to compare the record count of data which is one of the part of data reconciliation

Whenever we are doing data migration we have to building manifesto file which will have records of when, where and how much data got transferred. Here In this blog we are going to do manual data reconciliation using this manifesto file with these steps before starting reconciliation you have to migrate the data to BigQuery

• We will create manifesto table in BigQuery using terraform,

• Load manifesto CSV data to BigQuery using python script

• Compare Manifesto record count to BigQuery record count data

1. Create a manifesto table using terraform

Here we are going to use the same dataset which we have in created previous blog Create a BigQuery dataset and table using terraform we will be deploying our manifesto table in the same dataset by using dataset id

main.tf

main.tf

variable.tf

variable.tf

terraform.tfvars

terraform.tfvars

2. Load manifesto data from CSV file to BigQuery using python

Here python script we can use to load data from a CSV file in the GCP bucket

load data python script

In this python script need to add your project id and your bucket name

3. Compare BigQuery table record count to manifesto record count

compare record count using SQL

We need to run this script on BigQuery

Output

After running the SQL query

Record count output

Conclusion

In this blog, we have created a BigQuery manifesto table, load CSV data to a table and then SQL query to compare the manifesto record to the BigQuery table. Here you can get all the scripts from the GitHub account and an Excel sheet from google Sheets.

Reference

Google Docs

Loading CSV data from Cloud Storage

Links

tbl_manifesto Sheet1

GitHub - sudovazid/manifesto: This is terraform for manifesto file

This is series BigQuery 101. Here in this blogs I am going to build base of our project by creating BigQuery dataset and tables using terraform.

You can also find this code in GitHub

GitHub - sudovazid/gcp_terraform: Here is mostly terraform project avaiable to develop on Google…
*You can't perform that action at this time. You signed in with another tab or window. You signed out in another tab or…*github.com

Now before starting terraform code let’s understand dataset and table in BigQuery

As per google cloud a dataset is contained within a specific project. Datasets are top-level containers that are used to organize and control access to your tables and views

And in this blog we are going to use dataset to migrate store tables data and querying

In datasets folder we will have three terraform files main.tf, variable.tf, and output.tf

BigQuery dataset table folder

We will create main.tf, variable.tf, output.tf file on terraform folder In this main file we will have dataset resources to deploy BigQuery dataset, variable file will have variable which are going to be use in building dataset and output file to get dataset id after creating dataset which will be used to create tables

dataset.tf

Deploy dataset to BigQuery

variable.tf

output.tf

Now we will start with table folder

BigQuery table folder

Here we have main.tf and variable.tf

A BigQuery table contains individual records organized in rows. Each record is composed of columns (also called fields).

table.tf

variable.tf

Let’s combine all the resource with building modules.

BigQuery terraform module structure

Here we have main.tf to create terraform modules, variable.tf here we declare all the variables and terraform.tfvars here we pass the value in variables here we can also pass the provider variables :- project_id, region, zone

main.tf

variable.tf

terraform.tfvars

Conclusion

Here In this blog we have successfully deployed BigQuery dataset and table using terraform

References

Introduction to tables | BigQuery | Google Cloud
*A BigQuery table contains individual records organized in rows. Each record is composed of columns (also called…*cloud.google.com

Introduction to datasets | BigQuery | Google Cloud
*This page provides an overview of datasets in BigQuery. A dataset is contained within a specific project. Datasets are…*cloud.google.com

Trend Micro Cloud One has lots of products to secure our cloud, container and data centre. Which works for both enterprise data centres and in the cloud.

Here we are going to see setup file storage security on the S3 bucket if any user will upload the file into the bucket then trend micro FSS service scanned files and show the activity in the trend micro cloud one console and then we can transfer the cleaned file into promote bucket and malicious file into the quarantined bucket.

Prerequisite

Trend Micro cloud one access

AWS account admin access

Three Bucket for scan bucket, quarantined bucket and promote bucket

Setup

Login into Trend Micro Cloud Console Trend Micro Cloud One

Select File Security Storage

Click on Deploy

Select Scanner Stack and Storage Stack and Select us-west-2 (Oregon) region

Click on launch stack in AWS Account it will create a nested stack

Specify the scan bucket name in the S3BucketToScan parameter

After stack completion, we can see that we have two Stack Scanner Stack and Storage stack.

Copy and paste ScannerStackManagementRoleARN in trend micro console Deploy All-in-One-Stack Dialog box

Then add storage stack in trend micro-console, click on Add Storage

Copy and paste StorageStackManagementRoleARN in the trend micro console Deploy Storage Stack Dialog box

Trend Micro Cloud One Scan Activity

Step to setup Post scan Action Plugin

After completing Storage and Scanner stack we need to create a function to place clean files in one bucket and malicious files in another

Click on create a link to build lambda function stack ‘serverlessrepo-cloudone-filestorage-plugin-action-promote-or-quarantine’ name

Copy ScanResultTopicARN from Storage Stack and paste it into the ScanResultTopicARN parameter

Specify, and in cloud formation stack parameter

Test the solution

Download the Malicious zip file from this link

Upload the Zip file into the Scanned bucket

Upload and clean the file in the scanned bucket

Now we can monitor Scan Activity in Trend Micro Console

Also, files are removed from the scanned bucket to the Quarantined bucket and Promote Bucket

Conclusion

Trend Micro has which we can use to secure our data in cloud and enterprise data centres. Such as here we have used a file storage security service to secure data in an S3 bucket. So any user can’t upload unwanted files into the bucket.

Reference

Trend Micro Docs

Sign in — File Storage Security | Trend Micro Cloud One™ Documentation

GitHub Source

cloudone-filestorage-plugins/post-scan-actions/aws-python-promote-or-quarantine at master · trendmicro/cloudone-filestorage-plugins (github.com)

cloudone-filestorage-cloudformation-templates/FSS-All-In-One.template at master · trendmicro/cloudone-filestorage-cloudformation-templates (github.com)