Browsing: Pipeline

joining a new company as a data engineer. You inherit quite a few ETL pipelines and you are responsible for maintaining them. What do you think…

def make_rag_chunks(rows, max_chars=700): chunks = [] for row in rows: text = ( row.get(“text_preview”) or row.get(“rendered_text”) or row.get(“description”) or “” ) text = normalize_text(text) if not…

import os, warnings import numpy as np import pandas as pd import matplotlib.pyplot as plt warnings.filterwarnings(“ignore”) pd.set_option(“display.width”, 160) pd.set_option(“display.max_columns”, 30) print(“numpy:”, np.__version__) import scipy; print(“scipy:”, scipy.__version__)…

While recent breakthroughs in AI reasoning have largely been driven by massive scale, pouring in billions of parameters to cross complex cognitive thresholds—VibeThinker-3B is charting a…

, I mentioned that scheduling is the next wall I’ll be walking toward. So I guess, here I am, walking towards it But before I get…

def create_demo_image(path): img = Image.new(“RGB”, (320, 180), “white”) draw = ImageDraw.Draw(img) draw.rectangle([20, 20, 300, 160], outline=”black”, width=3) draw.ellipse([55, 45, 145, 135], outline=”black”, width=4) draw.line([180, 140, 285,…

fig, ax = plt.subplots(2, 2, figsize=(14, 9)) lang_counts.head(12).iloc[::-1].plot.barh(ax=ax[0, 0], color=”#76b900″) ax[0, 0].set_title(“Top 12 languages (sample)”); ax[0, 0].set_xlabel(“files”) df[“ext”].value_counts().head(12).iloc[::-1].plot.barh(ax=ax[0, 1], color=”#5b8def”) ax[0, 1].set_title(“Top 12 file extensions (sample)”);…

print(“\n” + “=”*70 + “\nPART 4: NDCG@10 evaluation\n” + “=”*70) eval_set = [ {“query”: “Where is most ATP produced in the cell?”, “rels”: {0: 2, 2:…

EXTRACT_PATS = [ r”\\boxed\{([^{}]+)\}”, r”final\s+answer\s*[:=]\s*([^\n]+)”, r”answer\s*[:=]\s*([^\n]+)”, ] def extract_final(text): if not text: return “” for p in EXTRACT_PATS: m = re.search(p, text, flags=re.IGNORECASE) if m: return…

of my data engineering journey series. In part one, I shared my 12-month roadmap for transitioning from data analyst to data engineer. This is where the…