Software Engineer

About the Position

We're seeking a Software Engineer with strong data engineering and AI systems expertise to build an AI-powered platform for production AI agents. This role involves building the infrastructure that evaluates, monitors, and optimizes AI agents at scale.

Responsibilities

• Developing online and offline evaluation frameworks for LLM agents using benchmark datasets, ground-truth labels, human feedback, and experimental results.
• Implementing automated validation pipelines that verify changes to prompts, models, context, and agent behavior before deployment.
• Investigating large-scale execution logs and agent traces to uncover quality issues, performance bottlenecks, reliability problems, and opportunities to reduce inference costs.
• Designing and maintaining data solutions built on analytical databases and column-oriented storage technologies such as BigQuery, ClickHouse, or comparable platforms.
• Building systems for long-term storage, replay, and analysis of production agent activity to support debugging and continuous improvement.
• Creating observability capabilities, including monitoring, trace inspection, logging, dashboards, and debugging tools for AI services.
• Contributing directly to the backend platform and agent ecosystem by enhancing existing AI agents and developing new capabilities when required.

Requirements

• Solid experience developing backend applications in Python or similar server-side technologies.
• Confident writing advanced SQL queries and working with large-scale datasets.
• Experience in deploying and maintaining cloud-based applications, preferably on Google Cloud Platform.
• Experience designing and implementing data pipelines, ETL/ELT processes, streaming systems, or production feedback loops.
• Familiarity with analytical databases, data warehouses, columnar storage solutions, and processing high-volume event or trace data.
• Strong understanding of distributed systems, observability, monitoring, logging, debugging, and the trade-offs involved in building reliable software.
• Ability to quickly navigate complex codebases and gain a deep understanding of existing architectures.
• Senior engineering judgment by making sound architectural decisions, evaluating technical trade-offs, and building solutions that are easy to maintain and extend.
• Ability to work in ambiguous environments, approach problems from first principles, and enjoy building infrastructure that powers production AI systems.

Nice to Have

• Hands-on experience developing infrastructure for LLM-powered applications or AI agents, including prompt optimization, model selection, context management, or inference optimization.
• Experience analyzing production traces from distributed applications.
• Building internal developer platforms or tooling for engineering, operations, or business teams.
• Experience with workflow orchestration platforms such as Temporal, Airflow, or similar technologies.
• Familiarity with highly regulated industries such as finance, compliance, risk, or audit.
• Previous experience working in an early-stage startup or another fast-paced engineering environment.

Location

Poland (Remote)

Employment Type

---

Experience Level

Experienced