Show HN: Gentrace – evaluation and observability for generative AI https://ift.tt/az1chgJ

Show HN: Gentrace – evaluation and observability for generative AI Hi HN, Gentrace is our new evaluation and observability tool for generative AI (open beta). Generative pipelines are hard to evaluate because outputs are subjective. Lots of developers end up just doing “gut checks” on a few inputs before shipping changes, or they build up a spreadsheet of test cases that they manually run through the pipeline. Some companies outsource filling out the spreadsheet. However, in any of these cases, you end up with a very slow and expensive process for evaluation. At one point, we did this too. Gentrace is the result of a pivot; it was an internal tool we used to automatically grade new PRs as developers shipped changes to generative pipelines that other people thought might be useful. Gentrace makes pre-production testing of generative pipelines continuous and nearly instantaneous. In Gentrace, you: - Import and/or construct suites of test data - Use a combination of AI and heuristic evaluators to grade for quality, hallucination, safety, etc - Use our interface to correct automated grades or add your own (yourself or a member of your team) Gentrace integrates at a code level for evaluation, meaning we test your generative AI pipeline the way you would test normal code. This allows you to test more than just prompt changes; for example, you can compare models (eg Claude 2 vs GPT-4 vs GPT 3.5 vs Llama 2) or see the effects of additional chained steps (”Rewrite the previous answer in the following tone:”). Here’s a video overview that goes into a bit more detail: https://youtu.be/XxgDPSrTWIw In production, Gentrace observes for speed, cost, and data flow. It also shows real user feedback as well. We do this by integrating via our SDK at a code level; Gentrace does not proxy requests. Soon, we’ll allow you to convert production data into test cases, allowing customer support to turn bad production generations into “failing tests” for AI teams to make pass. We process interim steps and multiple outputs as well, helping evaluate agent flows / chains where the “last output” isn’t always the only thing that matters. There’s been a lot of observability tools published recently. We differ from those by focusing more strongly on blending observability with strong evaluation and by using an SDK rather than a “man-in-the-middle” approach to capturing data (ie Gentrace can be down and your request to OpenAI will still succeed). Within the evaluation landscape, we differentiate by integrating with code (see above for benefits) for capturing generative outputs and by providing a customizable UI workflow for building evaluators. In Gentrace, you start with off-the-shelf automated evaluators and then customize them to your specific task. You also build and run new evaluators on old generative outputs. Finally, you easily override automated evaluators and/or blend automated evaluation with evaluation by humans on your team. We also focus on being suitable for business use. We are SOC 2 Type 1 compliant (Type 2 coming shortly), have robust legal documentation around data processing, security, and privacy, and have already passed several vendor legal and security reviews at large technology companies. Our standard usage-based pricing is available on the website: https://ift.tt/YcCg6LK If you are building features with generative AI, we would love to get your feedback. You can self-serve sign up (without a credit card) for a 14 day trial here: https://gentrace.ai/ We’re available right here for feedback and questions. We’re also available at support@gentrace.ai. Best, Doug, Vivek, and Daniel https://gentrace.ai August 23, 2023 at 10:08PM

Show HN: Script to Auto-Generate Commit Messages with AI https://ift.tt/bCT01ps

Show HN: Script to Auto-Generate Commit Messages with AI The goal is to integrate the capabilities of OpenAI's GPT-3 model into the Git commit process. The tool inspects the staged changes and auto-generates descriptive commit messages which can be used as template for the commit command. https://ift.tt/7NsIF4S August 23, 2023 at 06:32PM

Show HN: Pip install inference, open source computer vision deployment https://ift.tt/VKcGLdX

Show HN: Pip install inference, open source computer vision deployment Deploying vision models is time consuming and tedious. Setting up dependencies. Fixing conflicts. Configuring TRT acceleration. Flashing (and re-flashing) NVIDIA Jetsons. A streamlined, developer-friendly solution for inference is needed. We, the Roboflow team, have been hard at work open sourcing Inference, an open source vision deployment solution. Our solution is designed with developers in mind, offering a HTTP-based interface. Run models on your hardware without having to write architecture-specific inference code. Here's a demo showing how to go from a model to GPU inference on a video of a football game in ~10 minutes: https://www.youtube.com/watch?v=at-yuwIMiN4 Inference powers millions of daily API calls for global sports broadcasts, one of the world’s largest railways, a leading electric car manufacturer, and multiple other Fortune 500 companies, along with countless hackers’ hobby and research projects. Inference works in Docker and supports CPU (ARM and x86), NVIDIA GPU, and TRT. Inference manages dependencies and the environment. All you need to do is make HTTP requests to the server. YOLOv5, YOLOv8, YOLACT, CLIP, SAM, and other popular vision models are supported (some models need to be hosted on Roboflow first, see the docs; we're working on bring your own model weights!). Try it out and tell us what you think! https://ift.tt/OKAsDgU August 23, 2023 at 04:34PM

Show HN: To mute words on Twitter in batches, I created a browser extension https://ift.tt/k53Mv7x

Show HN: To mute words on Twitter in batches, I created a browser extension Hello HN community :) For a while now I've been watching my Twitter feed fill up with more and more crap. The fact that it's now possible to monetise your tweets has made things even crappier. At the same time, adding words to mute is a pain. First you have to find the option, buried deep in Twitter's settings, and then you have to add them one by one. So I came up with the idea of writing a little browser extension that lets you add lists of muted words, which can come from more or less anywhere, as long as they're in a certain format (a multi-line text file or a JSON array). The extension doesn't collect any data, you don't need an account to use it, and you can store your lists anywhere (although I'd recommend Github gist or Pastebin). To be honest, I made this mainly as an exercise to learn how a browser extension works and how to make one, but if it's useful to people, that's great! Feel free to let me know what you think and give it a try. If there's ever an audience for it, I'm thinking of taking it further by adding lists of people to mute, extracting your existing list of muted words from Twitter, etc.If you have any other ideas for development, I'd love to hear from you. https://ift.tt/tpSQiEL August 23, 2023 at 05:13PM

Show HN: Open-Source Chat AI Platform with Custom Knowledge https://ift.tt/mJPbUcQ

Show HN: Open-Source Chat AI Platform with Custom Knowledge https://ift.tt/1bXQscl August 22, 2023 at 09:38PM

Show HN: Scribble with AI – Create social media creatives with AI https://ift.tt/mnl6G7w

Show HN: Scribble with AI – Create social media creatives with AI I've been working on a project that will create social media creatives from prompts. This will let users create social media posts quickly. Let me give an example. Suppose I want to make a post on "Give me 8 health benefits of Ginger", it gives me 8 health benefits in the form of images that I can post on my social media. Here's the output -> https://ift.tt/XTrIbdt... If you'd like to try it out with your own prompts you can signup and give it a try. No credit cards is required to get started. I would love to have your feedback about this product. I am also looking for someone who might be interested in this project and is open to mentoring, so that I can learn to market this product and build a business around it. You can reach me out at scribblewithai@gmail.com Thanks! https://ift.tt/FYO4hXc August 23, 2023 at 01:06AM

Show HN: A CLI tool and controller to access K8s network through WireGuard https://ift.tt/375yvcW

Show HN: A CLI tool and controller to access K8s network through WireGuard Hi everyone! k8s-insider is a CLI tool to manage VPN networks and access Kubernetes workload and service resources directly through a WireGuard tunnel. It started as a simple Helm chart with a linuxserver/wireguard-like container image but, possibly due to my neurodivergence getting the better of me, evolved into a toolbox that, depending on the cluster configuration, can more or less automagically configure multiple networks with dynamic IP assignments and netpol-limited access to the cluster. I’ve found it to be a perfect middle ground between juggling multiple port-forwards and deploying something like Tailscale to the cluster. And it’s written in Rust, which made it oh-so-lovely to work on. :> https://ift.tt/Iuhm3zc August 22, 2023 at 09:00PM

Show HN: Breaklist – A Morning Briefing Printed on a Tiny Thermal Receipt https://ift.tt/z8A2kq6

Show HN: Breaklist – A Morning Briefing Printed on a Tiny Thermal Receipt I built Breaklist to organize the essential info I need to start my day. It generates a personalized morning briefing, optimized for thermal printers. The morning report currently includes: Task list Reminders Weather forecast Summary of latest top articles on Hacker News The result is a tidy, receipt-sized report that I can print and take on-the-go each morning. https://ift.tt/jlZ0kJo August 22, 2023 at 05:28PM

Show HN: Convert Research Papers into Dynamic Mind Maps with Claude https://ift.tt/sMXxWlk

Show HN: Convert Research Papers into Dynamic Mind Maps with Claude https://ift.tt/W24NCFB August 22, 2023 at 02:05PM

Show HN: Speleomorph, Shapeshifting Puzzle Metroidvania https://ift.tt/4oEkh2b

Show HN: Speleomorph, Shapeshifting Puzzle Metroidvania This was my entry for a Metroidvania Month game jam last year. I like how Metroidvanias make you rethink movement as the game goes on, and since I didn't want to design combat I focused on that. But I didn't want the different mechanics to be objects you pick up. The theme for the jam was "shapeshifting", which suggested the unusual mechanic seen here. Fortunately I'd forgotten about Snakebird or I might not have been bold enough to make the gravity+snake parts so prominent. And in the year since this came out, there have been at least three cool puzzle games nearby in design space: - Tetronimeow https://ift.tt/YQbHGkC has a heavy focus on rotation (which I skipped to avoid confusion), it gets much more Metroidvania mileage out of the central idea - Growmi https://ift.tt/TDCf3lZ does a great job building mechanics out of the snake+gravity idea - The Plumber Thing https://ift.tt/vyIGZDV If this looks familiar, I did post a Show HN last year, but dang invited me to re-submit it. I kept putting it off in hopes I could make a big splash when launching a new version: I have some ideas for touchscreens (currently keyboard only) and for an earlier game with more restricted abilities (no free transformation, then swapping with just one previous shape). But it's looking unlikely that I'll revisit it. https://ift.tt/IG9CtXx August 22, 2023 at 08:41AM

Show HN: VisionScript, abstract programming language for computer vision https://ift.tt/WQhvpjU

Show HN: VisionScript, abstract programming language for computer vision Hello! I'm James and I am working on VisionScript. With VisionScript, I want to empower people -- including everyone without any prior programming experience -- to build cool apps with vision. This weekend, I recorded a demo for VisionScript, in which I made apps that count how many cats are in an image and hides people in a video. Each app was < 10 lines of code. https://vimeo.com/856043804 VisionScript is built for the 10 year old inside of me who would have loved more visual programming languages with which to play. I want to show people the potential of programming and how you can make what you want with computers, whether it be a game that counts cats or an app that monitors how many birds flew past a tree. Those "wow" moments should come as soon as possible in one's learning experience. VisionScript is in active development. I started work on this project in July. There will likely be bugs; this is a passion project. Inspiration comes from Wolfram and Python. Behind the scenes, I am adopting what I am calling "lexical inference", which is to say there is a last state value on which functions infer; the language manages types and state. https://github.com/capjamesg/visionscript August 22, 2023 at 12:18AM

Show HN: Blast, A Load generator for TCP servers https://ift.tt/bR7lMGP

Show HN: Blast, A Load generator for TCP servers https://ift.tt/tZCf6de August 21, 2023 at 07:46PM

Show HN: DocXReducer – C# lib for docx size reducing without losses https://news.ycombinator.com/item?id=37208627

Show HN: DocXReducer – C# lib for docx size reducing without losses https://github.com/QCplus/DocXReducer August 21, 2023 at 06:04PM

Show HN: CryptexVault – Decentralized Password Manager https://ift.tt/DwgWRrt

Show HN: CryptexVault – Decentralized Password Manager Hi everyone! We started working on this project last year because a password manager that we used was being sold to another company and stopped offering their service. We were not happy with the alternatives that were available so we decided to give it a shot and build our own. We are now at a point where we have a working Beta and are looking for some feedback from the community. The main idea is to have data synchronization without a central server (single point of failure). When you create a Vault (inside which you can store your credentials and other sensitive data), you can "Link" it to other devices. By linking it to other devices, you can synchronize the data between them. - The Vault itself never leaves your device, it is encrypted and decrypted locally, and is only shared with the other devices that you link it to. The data synchronization is done using WebRTC, meaning, the data is transferred directly between the devices without going through a central server . If the devices cannot connect directly, the data is transferred through a TURN server (which is a server that is used to relay the data between the devices). - We are working on a way to allow the users to use their own TURN server (or a TURN server that they trust) or perhaps not use one at all, instead of using the one that we provide. Now, the only problem is how do your devices find each other - Using a signaling server. - We provide a signaling server that is used by default, but we're also working on a way to allow the users to use their own signaling server. In order to allow for easier testing, I've allowed the users to link devices without having to upgrade their account. I would appreciate any feedback you have for us, and if you have any questions, I'll be happy to answer them. Bl4cksmith https://ift.tt/PDmoVsR August 21, 2023 at 10:23AM

Show HN: Efficiently Query BigQuery and Snowflake from Postgres https://ift.tt/aBE9nM4

Show HN: Efficiently Query BigQuery and Snowflake from Postgres https://ift.tt/TOg23sW August 21, 2023 at 07:31AM

Show HN: Fake Hacker News – See what HN has to say before you post https://ift.tt/iy3mIab

Show HN: Fake Hacker News – See what HN has to say before you post Hi HN! I’ve been lurking for a while, but out of fear of being steamrolled by HN readers or maybe just natural introversion, I’ve always been too scared to post or comment. Which why 1. this is my first real Hacker News submission 2. my friend Michael and I built "Fake" Hacker News, a place to post and see what AI-generated HN comments might say. Here’s a video of me using fakeHN to test this very submission: https://ift.tt/gJoz8Pn?... And an example of one of our generated posts: https://ift.tt/CyszWid To try it, submit a title and text, and depending on traffic and the powers that be, after ~5 seconds, you’ll see some Fake HN comments and replies. We don’t support url submissions yet, but we’re happy to build it if the community wants it! Other features to knock out: deeply nested replies, streamed comments, and higher-fidelity comments mapping to real readers, since the generations now are still pretty shallow. Instead of the quick and dirty system in place now, we think it’d be really cool to see how more nuanced AI agents with the opinions and biases of real individual HN readers might respond. I’d love to see what fakeHN posts you’ve tried and hear any feedback, whether you feel like it’s more of a nifty toy or could eventually solve real problems. If nothing else, it’s been funny to try random posts and see the results. :) - Justin and Michael https://www.fakehn.com/ August 21, 2023 at 02:56AM

Show HN: Simple open-source tool for mocking using TS interfaces https://ift.tt/bMuUIHX

Show HN: Simple open-source tool for mocking using TS interfaces https://ift.tt/xKj4rTL August 21, 2023 at 02:23AM

Show HN: HTBL – A stupid way to write back ends in HTML https://ift.tt/oJSOl8q

Show HN: HTBL – A stupid way to write back ends in HTML https://ift.tt/2vsHo5Q August 21, 2023 at 12:35AM

Show HN: Cubisum – A number puzzle game inspired by Sudoku and Minesweeper https://ift.tt/b07ilIE

Show HN: Cubisum – A number puzzle game inspired by Sudoku and Minesweeper Hi HN, I'd like to share a logic number puzzle game I created! It is somewhat reminiscent of Sudoku and Minesweeper, but with its own twist. The rules are simple but it can be quite challenging! Please enjoy :) The feedback on the game prototype was invaluable and I incorporated a lot into what I hope is a more polished version of the game. What's new: Introduced a daily puzzle challenge. Dark mode. Enhanced user experience with improved UI. I'd love to hear your thoughts on these updates. Feedback is always appreciated! https://cubisum.com/ August 20, 2023 at 11:44PM

Show HN: Easy Sell AI – AI that sells for you on marketplace apps https://ift.tt/CNoWyKk

Show HN: Easy Sell AI – AI that sells for you on marketplace apps Hey Hacker News! I've developed Easy Sell AI to address the challenges I faced when I had to quickly sell items on platforms like Facebook Marketplace and OfferUp. Problem: 1. Constantly getting lowballed by potential buyers, and I just didn't know what I needed to say to get the price I wanted. 2. Spending hours negotiating with potential buyers, only to often get ghosted. 3. Encountering fake accounts and scammers, making the process even more tedious. Solution: Enter Easy Sell AI, an AI-powered tool that does all the selling for you on Marketplace Apps. With it: 1. The AI takes over negotiations, giving the perfect line to get you the price you deserve. 2. You're only notified when a genuine deal is on the table. 3. Set your preferences, like your minimum acceptable price and availability, and let Easy Sell AI manage the rest. Check it out and let me know your thoughts! https://easysell.ai August 20, 2023 at 07:54PM