With Helicone’s Customer Portal, you and your team can easily manage your customers and their usage. No need to handle authentication, build customer usage dashboards, implement billing systems, or keep track of your customer’s usage.

Helicone’s integration is quick and easy to set up and you can start collecting your customer’s usage and provide them with rich dashboards in minutes.

How does it work?

Helicone’s Customer Portal is a proxy that sits between your model and your customers.

Example:

Acme’s Integration of Helicone

Background: Acme offers an LLM chatbot and faces challenges with API management, billing, usage monitoring, authentication, rate limiting, and more.

Solution: Helicone

• Helicone Proxy: Serves as a secure gateway, authenticating client requests, enforcing rate limits, caching for performance, and logging for audit trails.

• Dashboard and Playground: Clients now access a Helicone-powered dashboard to monitor usage statistics, LLM logging, and experiment in a sandbox environment.

• Usage and Billing: Real-time API usage data is displayed via Helicone, ensuring accurate and transparent billing.

• CRM Integration: Client management is improved through a CRM system connected within Helicone, fostering better client relationships.

Take a tour

1. Getting started

Acme fills out our customer portal form. After being approved, they receive a top-level domain ‘acme.hconeai.com’ and a ‘customer portal’ button on their Helicone.ai dashboard.

2. Adding customers!

Once in the portal, Acme can add customers.

3. Authenticate the customer

Acme is providing its chatbot to customers. So, they create a new customer profile, with LLM usage limits and their ‘Provider Key’.

The ‘Provider Key’ can be anything you want, it is just a way to authenticate with the LLM provider (or your own model). Customers will never see the ‘Provider Key’ they will only see the ‘Helicone Keys’ they generate.

4. Manage customers

Acme can now manage all of its customers from the customer portal.

Acme then gives their customers access to Helicone by adding them as a member!

5. Customer’s Experience

Once their customers log in, they get access to the full Helicone feature set!

The customers can create ‘Helicone Keys’ which will be used as a Bearer token when making requests. They will map to the provided ‘Provider Key’ to authenticate with Acme’s model.

6. How customers will use your model

The customer can now make a request using the newly generated ‘Helicone Key’. They will be authenticated in Helicone and proxied to the Acme model endpoint.

 curl --request POST \
     --url https://acme.hconeai.com/v1 \
     --header 'Authorization: Bearer <<HELICONE_CUSTOMER_PORTAL_KEY>> \
     --data '{...}'

That’s it! Now your customers can use your model and you can charge them for it! Selling models has just become 10x easier.

If you are interested in getting started, please fill out our customer portal form.

"Clickhouse, like celestial beacons in the cosmic expanse, illuminate the path, signaling alerts in the vast ocean of real-time data." - Galileo Galilei

The Context

Helicone is an at-scale LLM monitoring platform (with tons of amazing features). Recently, we’ve been receiving a surge of requests for alerting, especially for cost, error rate, and latency. In this blog, we delve into the details of our design and build process with the rationale behind our approach.

Requirements

Approaching requirements at a startup requires a different mindset than you might be used to at larger, more established companies. It is critical to prioritize rapid development to find product-market fit. Perfection can wait. Below is a demonstration of the process we go through to determine the MVP.

1. Versatile Alerts: Support different alert types

   1. Let's begin with error rate alerts and determine if there's user demand for them before we branch out into more types.

2. Real-time push-based: Instant notification

   1. Near real-time pull-based works

3. Efficient Data Management: Store logs in minute blocks

   1. All in Clickhouse <3

4. Smart Alert Resolution: Avoid constant re-triggering

   1. Necessary for MVP or alerts could become useless. 5-minute cooldown before an alert can be resolved after it goes below the threshold.

5. Easy Integration: Easily add new types

   1. Meh, figure this out later

6. Scalable: Handle all the scale!

   1. As long as it can handle a few months of scale we’re good. If alerts become a hit, then it’s time to scale it up! We can always throw money at this problem and scale vertically

7. Customizable Thresholds: Threshold to signal alert

   1. Custom thresholds are enabled as some users expect a certain % of errors, which is a necessity.

8. Customizable Time Windows: Timespan to check the alert threshold

   1. Custom time windows are not a necessity, but easy to implement…

9. Customizable Minimum Request Threshold: Minimum requests that are required before an alert can be triggered

   1. Necessary to prevent low-traffic errors from triggering alerts, avoiding the need for higher threshold adjustments in high-traffic periods.

10. Multi-channel Notifications: SMS, email, webhooks, etc

    1. Email works

11. Real-Time Dashboards: Instantly be aware of the alert

    1. Near real-time seems to do the trick

Implementation Details

Configured alerts will simply be stored in Postgres, however, we need a solution for checking the alert state (triggered, resolved, cooldown). We compared the following 3 options:

Postgres

Postgres, while reliable, falls short in scaling, especially for time-sensitive metric aggregation.

Durable Objects

A good fit with our Cloudflare Workers stack but adds complexity in managing alerts across both Postgres and Durable Objects.

Clickhouse

Our choice. We currently log requests/responses to Clickhouse to support aggregations. We can leverage Clickhouse by adding a cron job that runs every minute to check the alert state. The max time window is 1 hour, so it will be quite efficient. If we hit a scaling bottleneck? We’re ready to scale financially (vertically) and technically as needed. Cue the all-nighters with Red Bull and coffee (not sponsored).

Postgres Table Definitions For Alerts & Alert History

Some denormalization takes place for faster querying.

CREATE TABLE alert (
    id uuid not null default gen_random_uuid() primary key,
    org_id UUID NOT NULL REFERENCES public.organization(id),
    metric TEXT NOT NULL,
    threshold DECIMAL NOT NULL,
    time_window BIGINT NOT NULL,
    time_block_duration BIGINT NOT NULL DEFAULT 60000,
    emails TEXT [] NOT NULL,
    status TEXT NOT NULL DEFAULT 'resolved',
    name TEXT NOT NULL,
    soft_delete boolean not null default false,
    created_at TIMESTAMP WITH TIME ZONE NULL DEFAULT NOW(),
    updated_at TIMESTAMP WITH TIME ZONE NULL DEFAULT NOW()
);

CREATE TABLE alert_history (
    id uuid not null default gen_random_uuid() primary key,
    org_id UUID NOT NULL REFERENCES public.organization(id),
    alert_id UUID REFERENCES alert(id) NOT NULL,
    alert_start_time TIMESTAMP WITH TIME ZONE NOT NULL,
    alert_end_time TIMESTAMP WITH TIME ZONE,
    alert_metric TEXT NOT NULL,
    alert_name TEXT NOT NULL,
    triggered_value TEXT NOT NULL,
    status TEXT NOT NULL,
    soft_delete boolean not null default false,
    created_at TIMESTAMP WITH TIME ZONE NULL DEFAULT NOW(),
    updated_at TIMESTAMP WITH TIME ZONE NULL DEFAULT NOW()
);

Yes, RLS is enabled.

Initialization: AlertManager Class

export class AlertManager {
  COOLDOWN_PERIOD_MS = 5 * 60 * 1000;
  private utilityKv: Env["UTILITY_KV"];
  private resendApiKey: Env["RESEND_API_KEY"];

  constructor(private alertStore: AlertStore, private env: Env) {
    this.utilityKv = env.UTILITY_KV;
    this.resendApiKey = env.RESEND_API_KEY;
  }
  // ...

This class is the heart of our alerting logic. It initializes with essential configurations, such as cooldown periods KV, and API keys.

The Core Function: checkAlerts

1. Fetch alerts from Postgres

const { data: alerts, error: alertsErr } = await this.supabaseClient
      .from("alert")
      .select("*")
      .eq("soft_delete", false);

2. Retrieve the alert’s current state from Clickhouse (error count, total count, etc)

SELECT
    COUNT() AS totalCount,
    COUNTIf(status BETWEEN 400 AND 599) AS errorCount
  FROM response_copy_v3
  WHERE 
    organization_id = {val_0: UUID} AND
    request_created_at >= toDateTime64(now(), 3) - INTERVAL {val_1: Int64} MILLISECOND

This query calculates the total count of events and counts the number of error events (status between 400 and 599) for a specific organization within a given time window.

Alert States

The AlertState object obtained from the Clickhouse query is pivotal for our alerting logic. It looks like this:

type AlertState = {
  totalCount: number;
  errorCount: number;
};

Check For Alert State Updates

For each AlertState, we need to calculate if there is an update.

const rate = alertState.totalCount > 0 ? (alertState.errorCount / alertState.totalCount) * 100 : 0;

Based on the rate and the alert state, the system marks alerts as 'triggered', 'resolved', or 'unchanged'. Here is the logic:

Check For Alert State Updates Code:

// Function to determine the update needed for an alert's state
async getAlertStateUpdate(alert, alertState, timestamp) {
  // Calculate the error rate
  const rate = alertState.totalCount > 0 ? (alertState.errorCount / alertState.totalCount) * 100 : 0;

  // Check if the error rate is below the threshold
  if (rate < alert.threshold) {
    // Handle scenario when the error rate is below the threshold
    return this.handleRateBelowThreshold(alert, timestamp);
  } else {
    // Handle scenario when the error rate is above or equal to the threshold
    return this.handleRateAboveThreshold(alert, rate, timestamp);
  }
}

// Handle alerts when the error rate is below the threshold
async handleRateBelowThreshold(alert, timestamp) {
  // If alert is already resolved, no change is needed
  if (alert.status === "resolved") {
    return { status: "unchanged", timestamp, alert };
  }

  // Get the start time of the cooldown period
  const cooldownStart = await this.getCooldownStart(alert.id);

  // If cooldown hasn't started, begin the cooldown period
  if (!cooldownStart) {
    await this.setCooldownStart(alert.id, timestamp);
    return { status: "unchanged", timestamp, alert };
  }

  // If cooldown has elapsed, mark the alert as resolved
  if (timestamp - cooldownStart >= this.COOLDOWN_PERIOD_MS) {
    await this.deleteCooldown(alert.id);
    return { status: "resolved", timestamp, alert };
  }

  // If cooldown is still active, no change is needed
  return { status: "unchanged", timestamp, alert };
}

// Handle alerts when the error rate is above or equal to the threshold
async handleRateAboveThreshold(alert, rate, timestamp) {
  // If alert was previously resolved, trigger a new alert
  if (alert.status === "resolved") {
    await this.deleteCooldown(alert.id);
    return { status: "triggered", timestamp, triggeredThreshold: rate, alert };
  }

  // If alert is already active, no change is needed
  return { status: "unchanged", timestamp, alert };
}

Handling Alert State Updates

If an alert is triggered

1. Update the alert records to be triggered

const { error: alertUpdateErr } = await this.supabaseClient
      .from("alert")
      .update({
        status: status,
        updated_at: new Date().toISOString(),
      })
      .in("id", alertIds);

2. Insert alert history records

 const { error: alertHistoryInsErr } = await this.supabaseClient
      .from("alert_history")
      .insert(alertHistories);

If an alert is resolved

1. Update the alert records to be resolved

const { error: alertUpdateErr } = await this.supabaseClient
      .from("alert")
      .update({
        status: status,
        updated_at: new Date().toISOString(),
      })
      .in("id", alertIds);

2. Update the alert history records, changing the status to resolved and adding the alert_end_time

const { error: alertHistoryUpdateErr } = await this.supabaseClient
      .from("alert_history")
      .update({
        status: status,
        alert_end_time: alertEndTime,
        updated_at: new Date().toISOString(),
      })
      .in("alert_id", alertIds)
      .eq("status", "triggered");

Send Emails

Send alert update emails to all users subscribed to the alert. For this, we use Resend.

The code is available here…

Conclusion

That’s it! Simple implementation and surprisingly scalable. These Clickhouse aggregation queries are incredibly quick. Consistently <0.05s to execute the query over the past 5/10 minutes. Running all alert checks asynchronously should not be an issue for a while.

What would I have done differently?

My first attempt at this was over-engineered. I went for push-based real-time alerts using Cloudflare Durable Objects. After realizing the extra complexity was unnecessary and would beget more complexity, I scrapped it and rewrote it in a day using Clickhouse. It was a painful lesson, but useful as I will now approach future projects with a different mindset.

“The diverse architectures and protocols of today's LLMs present unprecedented integration challenges, setting the stage for innovative solutions in AI observability.” - Nietzsche

The Problem: Fragmented LLM Ecosystem

In the AI industry, the explosion of large language models (LLMs) has led to a fragmented ecosystem. Different LLMs operate on distinct schemas and protocols, posing integration challenges for businesses seeking a unified AI observability solution. This fragmentation hinders seamless monitoring, analytics, and scalability.

Introducing Rosetta: Helicone's Solution

Helicone’s response to this challenge is Rosetta - an LLM-powered mapper generator. These mappers translate the diverse outputs and functionalities of various LLMs into a consistent schema that integrates seamlessly with Helicone's platform.

Core Components

Constructing this solution requires five key components:

• Unified and Executable Mapper Format

• Mapper Identification

• Intelligent Mapper Versioning

• Scalability & Speed

• Executable Code Security

Unified and Executable Mapper Format

Create a standard, versatile format to translate and execute mappings across diverse LLM data structures.

1. Sandbox Environment: Offers high security and isolation but is complex to implement.

2. JFilters: Simplifies processing but may struggle with complex JSON structures.

3. Pure JavaScript with Eval: Provides flexibility and adaptability; chosen for its ability to handle diverse data formats despite potential security concerns, mitigated by a strict approval process.

Mapper Identification

For identifying the appropriate mapper, we considered two main approaches:

1. Key-Based Identification: This straightforward method involves assigning unique keys, such as '/chat/completions', to each mapper.

2. Similarity Comparisons on JSON Bodies: A more advanced method that involves analyzing the JSON bodies for similarities to determine the appropriate mapper.

Ultimately, the decision to use key-based identification was driven by the need for a simple, efficient, and reliable system that can be easily managed and scaled as we grow.

Intelligent Versioning

Implement a robust versioning system to manage updates in mappers, accommodating new fields in either the output or input schema.

Key Extraction: Starts with extracting keys and JSON structure from the input.

Key Analysis: Identifies unmapped or 'ignored' keys (present in input but not in output schema).

Mapper Update: Updates existing mapper when new keys are detected, adding or ignoring them as needed.

Schema Monitoring: Continuously checks for changes in output schema through hash comparisons, triggering new mapper versions as required.

Scalability & Speed

With Helicone mapping the LLM outputs on read due to the rapidly evolving LLM schemas, both scalability and speed are crucial.

Design the architecture to be scalable, and capable of handling varying loads and complexities efficiently. To support scalability, it’s extremely important to not have to call an API to map, but instead to have the mapper in code or executed in code and retrieved from a cache system.

Executable Code Security

In Rosetta, each new mapper faces an approval process, crucial for two reasons. First, it addresses the inherent security risks associated with using eval for executing mappers. Second, it verifies the mapper's functionality, ensuring it operates as intended. If a mapper fails or encounters issues, our system automatically reverts to a previous, stable version, providing a reliable fallback mechanism.

Rosetta Mapper Lifecycle

Conclusion

Rosetta is Helicone's answer to the fast-changing world of large language models - a smart, secure bridge across the fragmented LLM landscape. So, what do you think - could Rosetta, as a standalone product, be the missing piece in your AI strategy? We'd love to hear if you're as excited about its potential as we are.

Introduction

The purpose of this article is to share some of the experiences and learnings that my cofounder, Justin Torre, and I have accumulated throughout our journey together working on Helicone.ai. Building any business is difficult, never mind a venture-backed startup with lofty expectations often placed on young individuals, and comes with inevitable challenges and bumps in the road. There will be days where you question your strategies, your product (very normal) and your yourself (even more normal). But at the end of the day, if you can answer the three questions below succinctly and clearly, then I genuinely believe that you’re well on your way.

• Do you care about the problem you’re solving?

• Do you care about the team you’re building with?

• Do you know your non-negotiables in life?

These questions align with your MISSION, VISION, VALUES and act as a compass whenever you are making critical decisions throughout your founder journey. Lets jump into it.

Do you actually care about the problem you’re solving?

Save the Total Adjustable Market (TAM) stuff for your investors and your future TechCrunch Disrupt presentation. It’s very easy to say you care about the problem or to say you know people who have this problem, but those aren’t answering the question - do you care about this problem? Have you personally experienced the pain-points? Do you have an emotional connection to the problem that goes deeper such as a female-founder deciding to change a part of the women’s health industry due to being let down in the past? Although founder-market fit isn’t the end all be all, its incredibly important and critical to be able to tell the story about why you’re working on the problem at hand to customers, investors, and other founders.

At Helicone, we sought out to build the best observability platform for Large-Language Models with the easiest integration on the market. We are a team of all developers creating a developer-tool that started out as a tool for ourselves. With the rise of LLM’s in web applications, we built out projects such as Valyr Chat, CourseGen, AIRapBattles, etc. and we needed a monitoring and logging tool for ourselves to debug, identify key cost areas, and segment requests. Our earliest customers were ourselves and close friends also building in the LLM-space, so the depth of caring for the problem we were addressing - there not being an easy, out-of-the-box monitoring tool - was extremely clear and strong for us.

Do you care about the team you’re building with?

One of the most common pieces of advice from Y Combinator was that founder relationships mattered a lot more than you think. And to absolutely no one’s surprise, founders (including ourselves) ignore this as we believe that over time a person’s merit and ability will trump all problems. Yes, it is good to index heavily on someone’s skill and ability regarding product, sales, or engineering. But during the early-stages of your company, you will quite literally be spending more than half of your day with your team. I’m not trying to discourage anyone from doing “Founder Dating” or YC’s own “Co-Founder matching”, but there needs to be a deeper connection besides “they are qualified to help me build my product”. Do they share similar values to you? Do they share a similar mission? And most importantly, do you enjoy working with them?

I met my co-founder, Justin, at Northeastern University where he was a Computer Science major and I had a Computer Science minor. We were partners in a class together junior year and we became best friends during and after the class. A builder at heart, Justin was a major driver for my own development in software engineering and over time we would hack on projects that eventually led us to getting into YC. I’m by no means saying that you have to be best friends with your co-founder, but there has to be a strong semblance of trust within your relationship with your co-founders. As YC always says, the most common reason why startups die is because of co-founder disputes and founders giving up.

Do you know your non-negotiables in life?

As hard as it is to get into Y Combinator (<1% acceptance rate for the W23 batch), it is even harder to build a successful, long-standing company. There are inevitably going to be days where you doubt and question every decision you and your team make and there are going to be even more days where you are slugging through hours upon hours of engineering work to get a feature out to a customer yesterday. To avoid founder burnout and to avoid resenting the path you’ve chosen, its extremely important to know what your non-negotiables in life are. These will help you stay sane as well as guide you through difficult decisions as a company.

Theres two sides to this question, one being what are things that you will make time for no matter what. Personally, I dedicate time in my week to playing and watching sports because it is a big part of who I am. Others in the Helicone team spend time building side-projects, going out to eat often, or exercising. Whatever it is that brings you genuine joy, you need to make time for it and nurture that part of who you are.

The other side of this question revolves around what you stand for as a company. For example, within the dev-tools space, there is quite a large variance in how much founders actually care about individual developers and hackers. Is your plan to go up market as soon as possible to secure sizable enterprise contracts? Or does being open-source and working closely with indie-hackers and builders sound enticing to you? These things aren’t mutually exclusive, but it’s important to know what your company and team are building towards and what are the things that you refuse to budge on.

Conclusion

We get asked question about Y Combinator all the time that are often something along the lines of:

• how do I get in?

• what should we say to the group partners during our interview?

• do you think this idea is good?

• is YC worth it?
  

Although its annoying, the answers to these questions are often not as clear as a one-word answer. I always end up asking the 3 questions I outlined above in response to better understand if the founder that I’m talking to has their heart and mind in the right place. There is an endless supply of ideas and a large supply of engineers/designers/sales people to work on those ideas. At the end of the day, it comes down to execution which involves having the right people to build, the right relationships to get buy-in, the right solution to a problem, and the right convictions to be able to be unwavering when confronted with a problem. Easier said than done, but if you can answer the 3 questions above clearly, you’ll be in a much better place than many other early-stage founders.

If you would like to talk about YC, Helicone, AI, or anything related to the startup space, please don’t hesitate to reach out to me at scott@helicone.ai. I’d love to help in any way that I can and I hope this letter helps you in your early founder journey!