NC Bench - A Benchmark for Creative Writing Models

This AI benchmark has been built by Novelcrafter (hence the name NC Bench) and is designed to evaluate the performance of creative writing models as well as various other related tasks.

The following categories are included in the benchmark:

  • Creative Writing: Assessing the model's ability to generate pleasing, well-structured text that adheres to good writing principles.
  • Instruction Following: Evaluating the model's capacity to accurately follow specific instructions.
  • Utility: Testing the model's ability to perform data extraction and reformatting tasks without hallucinations.
  • Tooling: Checking if the model can be interfaced with programmatic interfaces and its ability to produce error-free output.
  • Language: Assessing the model's proficiency in generating high-quality text across multiple languages.

Test Focus, AI ethics

Our focus is on enhancing the writing process with AI assistance rather than replacing it entirely.

This benchmark tests creative writing quality rather than a model's ability to write complete stories or replace the entire writing process. We believe AI should serve as a tool to assist writers, which is reflected in our test focus:

  1. Text Manipulation: Evaluating the model's ability to modify given text without introducing hallucinations. This is valuable for writers who need to change tenses, rephrase paragraphs, or make minor adjustments.
  2. Text Generation: Assessing the model's capacity to provide inspiration or ideas while closely following human-given instructions and maintaining coherence with the provided storyline.
  3. Text Summarization: Testing the model's ability to create concise elevator pitches or summaries, useful for quick overviews or marketing purposes.
  4. Text Translation: Evaluating the model's proficiency in translating text into other languages, enabling writers to reach broader audiences or draw inspiration from diverse linguistic sources.

In summary, we do not focus on replacing the full writing process, but rather on assisting writers in their writing process by providing specific tools that can help them with their work.