AI/ML News & Innovations Hub

@inproceedings{huidrom-etal-2026-aura,
    title = "Aura 💫: A Self-Hostable Platform for Human Evaluation of {NLG} for Under-resourced Languages and Beyond",
    author = "Huidrom, Rudali  and
      Nelson, Khangembam Alex D  and
      Khumukcham, Nicholas  and
      Pangsatabam, Hoomexsun  and
      L, Jesu Nazarene Roy  and
      Trivedi, Kanal  and
      Kumar, Vikas  and
      Das, Pinaki  and
      Singh, Kshetrimayum Boynao  and
      Bisht, Tarun  and
      Konjengbam, Justice  and
      Anand, Konjengbam  and
      Goyal, Vikram",
    editor = "Zhang, Ningyu  and
      Wijaya, Derry",
    booktitle = "Proceedings of the 15th International Joint Conference on Natural Language Processing and the 5th Conference of the {A}sia-Pacific Chapter of the {A}ssociation for {C}omputational {L}inguistics: System Demonstrations",
    month = nov,
    year = "2026",
    address = "Hengqin, China",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.aacl-demo.2/",
    pages = "9--16",
    ISBN = "979-8-89176-441-5",
    abstract = "Human evaluation is the gold standard for Natural Language Generation (NLG), but the tooling is fragmented: most platforms serve a single task or a single protocol, and none treat the growing practice of assisting annotators with an LLM as a measurable object of study. These gaps fall hardest on indigenous and under-resourced languages, where paid crowds are scarce, references are unavailable, and data governance matters. We present Aura, an open, self-hostable platform that unifies five evaluation protocols (Likert/DA, pairwise, ranking, error-span, post-editing) across eight task families under one segment schema and one result model, records writing-system and speaker provenance as first-class fields, and runs entirely offline. Its distinguishing feature is a grounded, advisory-only, logged LLM assistant paired with a built-in randomised experiment that estimates the effect of AI assistance on inter-annotator agreement and time-on-task. Aura{'}s agreement statistics match reference implementations to machine precision. We conducted a human evaluation experiment on three English{\textrightarrow}Indian-language ranking campaigns (N=9 annotators) which returned a mean usability rating of 4.06/5. Aura is released as open-source software."
}

<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="huidrom-etal-2026-aura">
    <titleInfo>
        <title>Aura 💫: A Self-Hostable Platform for Human Evaluation of NLG for Under-resourced Languages and Beyond</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Rudali</namePart>
        <namePart type="family">Huidrom</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Khangembam</namePart>
        <namePart type="given">Alex</namePart>
        <namePart type="given">D</namePart>
        <namePart type="family">Nelson</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Nicholas</namePart>
        <namePart type="family">Khumukcham</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Hoomexsun</namePart>
        <namePart type="family">Pangsatabam</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Jesu</namePart>
        <namePart type="given">Nazarene</namePart>
        <namePart type="given">Roy</namePart>
        <namePart type="family">L</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Kanal</namePart>
        <namePart type="family">Trivedi</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Vikas</namePart>
        <namePart type="family">Kumar</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Pinaki</namePart>
        <namePart type="family">Das</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Kshetrimayum</namePart>
        <namePart type="given">Boynao</namePart>
        <namePart type="family">Singh</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Tarun</namePart>
        <namePart type="family">Bisht</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Justice</namePart>
        <namePart type="family">Konjengbam</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Konjengbam</namePart>
        <namePart type="family">Anand</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Vikram</namePart>
        <namePart type="family">Goyal</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026-11</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 15th International Joint Conference on Natural Language Processing and the 5th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics: System Demonstrations</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Ningyu</namePart>
            <namePart type="family">Zhang</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Derry</namePart>
            <namePart type="family">Wijaya</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Hengqin, China</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-441-5</identifier>
    </relatedItem>
    <abstract>Human evaluation is the gold standard for Natural Language Generation (NLG), but the tooling is fragmented: most platforms serve a single task or a single protocol, and none treat the growing practice of assisting annotators with an LLM as a measurable object of study. These gaps fall hardest on indigenous and under-resourced languages, where paid crowds are scarce, references are unavailable, and data governance matters. We present Aura, an open, self-hostable platform that unifies five evaluation protocols (Likert/DA, pairwise, ranking, error-span, post-editing) across eight task families under one segment schema and one result model, records writing-system and speaker provenance as first-class fields, and runs entirely offline. Its distinguishing feature is a grounded, advisory-only, logged LLM assistant paired with a built-in randomised experiment that estimates the effect of AI assistance on inter-annotator agreement and time-on-task. Aura’s agreement statistics match reference implementations to machine precision. We conducted a human evaluation experiment on three English→Indian-language ranking campaigns (N=9 annotators) which returned a mean usability rating of 4.06/5. Aura is released as open-source software.</abstract>
    <identifier type="citekey">huidrom-etal-2026-aura</identifier>
    <location>
        <url>https://aclanthology.org/2026.aacl-demo.2/</url>
    </location>
    <part>
        <date>2026-11</date>
        <extent unit="page">
            <start>9</start>
            <end>16</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Aura 💫: A Self-Hostable Platform for Human Evaluation of NLG for Under-resourced Languages and Beyond
%A Huidrom, Rudali
%A Nelson, Khangembam Alex D.
%A Khumukcham, Nicholas
%A Pangsatabam, Hoomexsun
%A L, Jesu Nazarene Roy
%A Trivedi, Kanal
%A Kumar, Vikas
%A Das, Pinaki
%A Singh, Kshetrimayum Boynao
%A Bisht, Tarun
%A Konjengbam, Justice
%A Anand, Konjengbam
%A Goyal, Vikram
%Y Zhang, Ningyu
%Y Wijaya, Derry
%S Proceedings of the 15th International Joint Conference on Natural Language Processing and the 5th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics: System Demonstrations
%D 2026
%8 November
%I Association for Computational Linguistics
%C Hengqin, China
%@ 979-8-89176-441-5
%F huidrom-etal-2026-aura
%X Human evaluation is the gold standard for Natural Language Generation (NLG), but the tooling is fragmented: most platforms serve a single task or a single protocol, and none treat the growing practice of assisting annotators with an LLM as a measurable object of study. These gaps fall hardest on indigenous and under-resourced languages, where paid crowds are scarce, references are unavailable, and data governance matters. We present Aura, an open, self-hostable platform that unifies five evaluation protocols (Likert/DA, pairwise, ranking, error-span, post-editing) across eight task families under one segment schema and one result model, records writing-system and speaker provenance as first-class fields, and runs entirely offline. Its distinguishing feature is a grounded, advisory-only, logged LLM assistant paired with a built-in randomised experiment that estimates the effect of AI assistance on inter-annotator agreement and time-on-task. Aura’s agreement statistics match reference implementations to machine precision. We conducted a human evaluation experiment on three English→Indian-language ranking campaigns (N=9 annotators) which returned a mean usability rating of 4.06/5. Aura is released as open-source software.
%U https://aclanthology.org/2026.aacl-demo.2/
%P 9-16

Markdown (Informal)

[Aura 💫: A Self-Hostable Platform for Human Evaluation of NLG for Under-resourced Languages and Beyond](https://aclanthology.org/2026.aacl-demo.2/) (Huidrom et al., AACL-IJCNLP 2026)

ACL

Rudali Huidrom, Khangembam Alex D Nelson, Nicholas Khumukcham, Hoomexsun Pangsatabam, Jesu Nazarene Roy L, Kanal Trivedi, Vikas Kumar, Pinaki Das, Kshetrimayum Boynao Singh, Tarun Bisht, Justice Konjengbam, Konjengbam Anand, and Vikram Goyal. 2026. Aura 💫: A Self-Hostable Platform for Human Evaluation of NLG for Under-resourced Languages and Beyond. In Proceedings of the 15th International Joint Conference on Natural Language Processing and the 5th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics: System Demonstrations, pages 9–16, Hengqin, China. Association for Computational Linguistics.