Previous studies demonstrated that ultra-processed foods (UPF) affect overall diet quality. However, none have yet examined this relation across different age groups in Brazil. This study assessed the relationship between diet quality and the consumption of UPF in a Brazilian population according to age groups. This was a cross-sectional study that analysed food consumption data from 46 164 Brazilians aged ≥10 years who participated in the 2017–2018 National Dietary Survey. Food and beverages consumed were recorded by two 24-h recalls. All food items were classified as UPF or non-UPF according to the Nova system. Diet quality was evaluated using nutritional density and the prevalence of inadequate nutrient consumption, according to the quintiles of energy contribution of UPF. The association between diet quality and UPF consumption was evaluated by linear and Poisson regressions, with adjustment for sociodemographic variables, stratified by age groups (adolescents, adults and older adults). The consumption of UPF increased the densities of carbohydrates, free sugar, saturated fat and Na and decreased the densities of proteins, fibres and potassium in three age groups. Higher prevalence ratios (PR) of inadequate consumption of free sugar and fibre among the lower and higher quintiles of energy contribution of UPF among adolescents (PR = 2·02, 95 % CI = 1·82, 2·25; PR = 1·88, 95 % CI = 1·68, 2·10), adults (PR = 1·86, 95 % CI = 1·75, 1·98; PR = 1·70, 95 % CI = 1·60, 1·80) and older adults (PR = 1·48, 95 % CI = 1·30, 1·69; PR = 1·24, 95 % CI = 1·09, 1·40). UPF consumption was negatively associated with diet quality across different age groups. Thus, interventions targeting UPF consumption should be implemented across life stages to improve overall diet quality.