Skip to content

adding ability to control maximum size of rss #283

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 4 commits into
base: master
Choose a base branch
from
Open

adding ability to control maximum size of rss #283

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
678f331
adding ability to control maximum size of rss
vitalyveksler Mar 23, 2025
5d8272b
updating readyme
vitalyveksler Mar 23, 2025
6e9a7fa
updating error message
vitalyveksler Apr 4, 2025
e7ab45c
improving error handling
vitalyveksler Apr 4, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
260 changes: 1 addition & 259 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,262 +10,4 @@
[build-image]: https://github.com/rbren/rss-parser/workflows/tests/badge.svg
[build-link]: https://github.com/rbren/rss-parser/actions

A small library for turning RSS XML feeds into JavaScript objects.

## Installation
```bash
npm install --save rss-parser
```

## Usage
You can parse RSS from a URL (`parser.parseURL`) or an XML string (`parser.parseString`).

Both callbacks and Promises are supported.

### NodeJS
Here's an example in NodeJS using Promises with async/await:

```js
let Parser = require('rss-parser');
let parser = new Parser();

(async () => {

let feed = await parser.parseURL('https://www.reddit.com/.rss');
console.log(feed.title);

feed.items.forEach(item => {
console.log(item.title + ':' + item.link)
});

})();
```

### TypeScript
When using TypeScript, you can set a type to control the custom fields:

```typescript
import Parser from 'rss-parser';

type CustomFeed = {foo: string};
type CustomItem = {bar: number};

const parser: Parser<CustomFeed, CustomItem> = new Parser({
customFields: {
feed: ['foo', 'baz'],
// ^ will error because `baz` is not a key of CustomFeed
item: ['bar']
}
});

(async () => {
const parser = new Parser();
const feed = await parser.parseURL('https://www.reddit.com/.rss');
console.log(feed.title); // feed will have a `foo` property, type as a string

feed.items.forEach(item => {
console.log(item.title + ':' + item.link) // item will have a `bar` property type as a number
});
})();
```

### Web
> We recommend using a bundler like [webpack](https://webpack.js.org/), but we also provide
> pre-built browser distributions in the `dist/` folder. If you use the pre-built distribution,
> you'll need a [polyfill](https://github.com/taylorhakes/promise-polyfill) for Promise support.

Here's an example in the browser using callbacks:

```html
<script src="/node_modules/rss-parser/dist/rss-parser.min.js"></script>
<script>

// Note: some RSS feeds can't be loaded in the browser due to CORS security.
// To get around this, you can use a proxy.
const CORS_PROXY = "https://cors-anywhere.herokuapp.com/"

let parser = new RSSParser();
parser.parseURL(CORS_PROXY + 'https://www.reddit.com/.rss', function(err, feed) {
if (err) throw err;
console.log(feed.title);
feed.items.forEach(function(entry) {
console.log(entry.title + ':' + entry.link);
})
})

</script>
```

### Upgrading from v2 to v3
A few minor breaking changes were made in v3. Here's what you need to know:

* You need to construct a `new Parser()` before calling `parseString` or `parseURL`
* `parseFile` is no longer available (for better browser support)
* `options` are now passed to the Parser constructor
* `parsed.feed` is now just `feed` (top-level object removed)
* `feed.entries` is now `feed.items` (to better match RSS XML)


## Output
Check out the full output format in [test/output/reddit.json](test/output/reddit.json)

```yaml
feedUrl: 'https://www.reddit.com/.rss'
title: 'reddit: the front page of the internet'
description: ""
link: 'https://www.reddit.com/'
items:
- title: 'The water is too deep, so he improvises'
link: 'https://www.reddit.com/r/funny/comments/3skxqc/the_water_is_too_deep_so_he_improvises/'
pubDate: 'Thu, 12 Nov 2015 21:16:39 +0000'
creator: "John Doe"
content: '<a href="http://example.com">this is a link</a> &amp; <b>this is bold text</b>'
contentSnippet: 'this is a link & this is bold text'
guid: 'https://www.reddit.com/r/funny/comments/3skxqc/the_water_is_too_deep_so_he_improvises/'
categories:
- funny
isoDate: '2015-11-12T21:16:39.000Z'
```

##### Notes:
* The `contentSnippet` field strips out HTML tags and unescapes HTML entities
* The `dc:` prefix will be removed from all fields
* Both `dc:date` and `pubDate` will be available in ISO 8601 format as `isoDate`
* If `author` is specified, but not `dc:creator`, `creator` will be set to `author` ([see article](http://www.lowter.com/blogs/2008/2/9/rss-dccreator-author))
* Atom's `updated` becomes `lastBuildDate` for consistency

## XML Options

### Custom Fields
If your RSS feed contains fields that aren't currently returned, you can access them using the `customFields` option.

```js
let parser = new Parser({
customFields: {
feed: ['otherTitle', 'extendedDescription'],
item: ['coAuthor','subtitle'],
}
});

parser.parseURL('https://www.reddit.com/.rss', function(err, feed) {
console.log(feed.extendedDescription);

feed.items.forEach(function(entry) {
console.log(entry.coAuthor + ':' + entry.subtitle);
})
})
```

To rename fields, you can pass in an array with two items, in the format `[fromField, toField]`:

```js
let parser = new Parser({
customFields: {
item: [
['dc:coAuthor', 'coAuthor'],
]
}
})
```

To pass additional flags, provide an object as the third array item. Currently there is one such flag:

* `keepArray (false)` - set to `true` to return *all* values for fields that can have multiple entries.
* `includeSnippet (false)` - set to `true` to add an additional field, `${toField}Snippet`, with HTML stripped out

```js
let parser = new Parser({
customFields: {
item: [
['media:content', 'media:content', {keepArray: true}],
]
}
})
```

### Default RSS version
If your RSS Feed doesn't contain a `<rss>` tag with a `version` attribute,
you can pass a `defaultRSS` option for the Parser to use:
```js
let parser = new Parser({
defaultRSS: 2.0
});
```


### xml2js passthrough
`rss-parser` uses [xml2js](https://github.com/Leonidas-from-XIV/node-xml2js)
to parse XML. You can pass [these options](https://github.com/Leonidas-from-XIV/node-xml2js#options)
to `new xml2js.Parser()` by specifying `options.xml2js`:

```js
let parser = new Parser({
xml2js: {
emptyTag: '--EMPTY--',
}
});
```

## HTTP Options

### Timeout
You can set the amount of time (in milliseconds) to wait before the HTTP request times out (default 60 seconds):

```js
let parser = new Parser({
timeout: 1000,
});
```

### Headers
You can pass headers to the HTTP request:
```js
let parser = new Parser({
headers: {'User-Agent': 'something different'},
});
```

### Redirects
By default, `parseURL` will follow up to five redirects. You can change this
with `options.maxRedirects`.

```js
let parser = new Parser({maxRedirects: 100});
```

### Request passthrough
`rss-parser` uses [http](https://nodejs.org/docs/latest/api/http.html#http_http_get_url_options_callback)/[https](https://nodejs.org/docs/latest/api/https.html#https_https_get_url_options_callback) module
to do requests. You can pass [these options](https://nodejs.org/docs/latest/api/https.html#https_https_request_options_callback)
to `http.get()`/`https.get()` by specifying `options.requestOptions`:

e.g. to allow unauthorized certificate
```js
let parser = new Parser({
requestOptions: {
rejectUnauthorized: false
}
});
```

## Contributing
Contributions are welcome! If you are adding a feature or fixing a bug, please be sure to add a [test case](https://github.com/bobby-brennan/rss-parser/tree/master/test/input)

### Running Tests
The tests run the RSS parser for several sample RSS feeds in `test/input` and outputs the resulting JSON into `test/output`. If there are any changes to the output files the tests will fail.

To check if your changes affect the output of any test cases, run

`npm test`

To update the output files with your changes, run

`WRITE_GOLDEN=true npm test`

### Publishing Releases
```bash
npm run build
git commit -a -m "Build distribution"
npm version minor # or major/patch
npm publish
git push --follow-tags
```
This version is a patch onto rss-parser version 3.13.0 that enforces maximum size of the rss. Once our pull requests is approved, this package will no longer be required.
6 changes: 6 additions & 0 deletions lib/parser.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -111,6 +111,12 @@ class Parser {
res.setEncoding(encoding);
res.on('data', (chunk) => {
xml += chunk;

if (this.options.maxSize && xml.length > this.options.maxSize ) {
return reject(new Error("Feed exceeds allowed size"));
} else if (xml.length > this.options.maxSize) {
return reject(new Error(`Feed exceeds allowed size of ${this.options.maxSize / 1000000}MB`));
}
});
res.on('end', () => {
return this.parseString(xml).then(resolve, reject);
Expand Down
10 changes: 5 additions & 5 deletions package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "rss-parser",
"version": "3.13.0",
"name": "rss-parser-with-maxsize",
"version": "3.13.3",
"main": "index.js",
"types": "index.d.ts",
"scripts": {
Expand Down Expand Up @@ -32,12 +32,12 @@
},
"repository": {
"type": "git",
"url": "git+https://github.com/bobby-brennan/rss-parser.git"
"url": "git+https://github.com/webalgorithm/rss-parser.git"
},
"bugs": {
"url": "https://github.com/bobby-brennan/rss-parser/issues"
"url": "https://github.com/webalgorithm/rss-parser/issues"
},
"homepage": "https://github.com/bobby-brennan/rss-parser#readme",
"homepage": "https://github.com/webalgorithm/rss-parser#readme",
"description": "A lightweight RSS parser, for Node and the browser",
"keywords": [
"RSS",
Expand Down