Encoding ASCII up to 255

[gevro]

How do I use BString / BStringSet for values > 127?

PacBio data uses 0:255 ranges for fp/rp/fi/ri tags. I am trying to use BioStrings to work with these tags.

But I ran into trouble, for values > 127.

Technically, ASCII shoudl go up to 255, so I'm not sure why BString shouldn't be able to handle things like this:
BStringSet(rawToChar(as.raw(135)))

It seems to just return '?' for any value > 127.

[gevro]

Also, how do I pass ASCII (raw) value 0 to BStringSet?

This works:

> BStringSet(rawToChar(as.raw(70)))
BStringSet object of length 1:
    width seq
[1]     1 F

This doesn't work

> BStringSet(rawToChar(as.raw(0)))
BStringSet object of length 1:
    width seq
[1]     0 

[hpages]

Hi,

rawToChar(as.raw(0)) is an empty string so no surprise that BStringSet(rawToChar(as.raw(0))) returns what it does.

The '?' you see when displaying BStringSet(rawToChar(as.raw(135))) is only a displaying issue (more precisely it's a show() method issue), but the content of the BStringSet object is as expected:

x <- BString(rawToChar(as.raw(135)))
x
# 1-letter BString object
# seq: �

as.integer(x)
# [1] 135

You want to avoid using the character string intermediate representation when going from raw to BString. One way to do this is:

x <- as(as(as.raw(c(0L, 135L, 0L, 255L)), "XRaw"), "BString")

Again, display is broken:

x
# 4-letter BString object
# Error in XVector:::extract_character_from_XRaw_by_ranges(x, start, width,  : 
#   embedded nul in string: '\0\x87\0\xff'

But content is correct:

length(x)
# [1] 4

as.integer(x)
[1]   0 135   0 255

length(c(x, reverse(x)))
# [1] 8

as.integer(c(x, reverse(x)))
# [1]   0 135   0 255 255   0 135   0

2 things that would improve the current situation: (1) make it easier to go from raw to BString or BStringSet, and (2) fix display.

Unfortunately I don't have the resources to work on these improvements at the moment but PRs are welcome.

Thanks,
H.

[FelixErnst]

Hi. quite late to the conversation.

What the rational for these values? I am not familiar with what the values represent. I assume some kind of quality values.
In addition to the PR as suggest, you might want to consider a dedicate class as extension of Biostrings package.

In Modstrings, currently 166 integer values are used for representing RNA modifications. Internally, this these integers are used and just converted for 'show' to actual letters. Both parts are totally independent, so you are not constricted to a 1:1 ASCII to int conversion.

Felix

[gevro]

These ascii values are encoding kinetic information for PACbio sequencing data. they are a standard part of the Pacbio data format. So it would be useful to have ability to convert integers from 0 to 255.

[FelixErnst]

From experience, representing that with Biostrings should not be a problem. However, displaying these values to console is separate issue. What you in console, is not necessarily what is stored.

Example: For DNA, the values 1,2,4,8 (4 bits) are used to store letter G, A, T, C (at least 8 bit; in case of ASCII 7 bit). Only during display to console, does the conversion take place. The following snippet throws an error, but this is just because the lookup lkup is NULL

> dss <- DNAStringSet(c("G","A","T","C"))
> XVector:::extract_character_from_SharedRaw_by_positions(dss[[1]]@shared,1L,1L,collapse = FALSE)
Error in XVector:::extract_character_from_SharedRaw_by_positions(dss[[1]]@shared,  : 
  'x' contains an invalid byte (4 = char '�') at position 1

So you need to be careful when debugging your code, that you not mistake "data stored" and "representation shown".
Your example does actually work, because you put in an empty string.

> BStringSet(rawToChar(as.raw(0)))
BStringSet object of length 1:
    width seq
[1]     0 
# This is the same as this
> BStringSet("")
BStringSet object of length 1:
    width seq
[1]     0 
# Because zero for R is actually an empty character
> rawToChar(as.raw(0))
""

So you might have the problem, that without realizing, you did an integer to character conversion, which you actually don't want to do. You want to keep the value as is, if I understand you correctly. The BString is the big string class in Biostrings, which means "a big string (a long sequence of characters)" (quote from the man page ?BString)

I could image, that you need something more like this, since quality values can be stored using the XStringQuality class from Biostrings.

> qs <- QualityScaledDNAStringSet(x = DNAStringSet(c("G","A","T","C")),quality = PhredQuality(IntegerList(1,2,3,4)))
> qs
  A QualityScaledDNAStringSet instance containing:

DNAStringSet object of length 4:
    width seq
[1]     1 G
[2]     1 A
[3]     1 T
[4]     1 C

PhredQuality object of length 4:
    width seq
[1]     1 "
[2]     1 #
[3]     1 $
[4]     1 %

The interesting bit is, that you need to use the IntegerList class as far as I know, since you cannot create a XStringQuality from a list and using c which turns the values during BString construction into actual characters.

> il <- IntegerList(0:255)
> il
IntegerList of length 1
[[1]] 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 ... 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255
> pq <- PhredQuality(il)
> pq
PhredQuality object of length 1:
    width seq
[1]   256 !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijk...~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

However, also in pq not the integer values are stored, since during object creation the are converted as stated in the manual "PhredQuality objects store characters that are interpreted as [0 - 99] quality measures by subtracting 33 from their ASCII decimal representation (e.g. ! = 0, " = 1, # = 2, ...)"

> as.integer(pq[[1]])
  [1]  33  34  35  36  37  38  39  40  41  42  43  44  45  46  47  48  49  50  51  52  53  54  55  56  57  58  59  60  61  62  63  64  65  66  67  68  69  70  71
 [40]  72  73  74  75  76  77  78  79  80  81  82  83  84  85  86  87  88  89  90  91  92  93  94  95  96  97  98  99 100 101 102 103 104 105 106 107 108 109 110
 [79] 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126
[118] 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126
[157] 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126
[196] 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126
[235] 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126 126

So you might want to consider at least three steps:

• Implement your own XStringQuality. which does not convert any integer values.
• Check that you can still use the BStringSet class as backend to represent the values for your XStringQuality, Otherwise implement your own XStringSet class.
• figure out a meaningful way, how to print those values to console.

Personally, I would start from the last step.

[FelixErnst]

These ascii values are encoding kinetic information for PACbio sequencing data. they are a standard part of the Pacbio data format. So it would be useful to have ability to convert integers from 0 to 255.

Hint: this is a bit confusing. Do you want to store ASCII or integer values from 0 to 255? It is either or. Both cannot be true at the same time.

ASCII also just has 128 values and only some are human readable: https://en.wikipedia.org/wiki/ASCII

[gevro]

Sorry, I by mistake wrote ASCII. What I mean is simply integer values from 0 to 255.

Here is an example tag from a read in a BAM file:
ip:B:C,30,29,31,34,34,34,27,28,27,30,28,31,29,46,36,29,29,38,27,33,33,30,34,27,39,26,37,46,34,27,29,29,34,32,33,36,34,29,32,30,25,28,30,30,34,31,28,31,33,25,37,34,22,27,32,45,26,23,25,26,25,30,35,30,28,31,33,37,23,28,31,41,30,27,25,30,29,30,30,40,30,29,31,28,31,45,67,29,34,34,36,23,27,25,33,36,35,33,27,33,30,30,26,27,26,38,82,26,36,33,47,28,25,30,34,32,31,38,27,24,42,29,30,27,28,25,31,26,30,36,25,26,27,33,31,34,49,32,32,28,24,24,33,27,26,26,34,31,30,29,29,22,25,29,30,25,39,32,29,28,26,26,27,42,28,32,28,28,32,34,23,28,28,31,34,32,34,31,39,28,39,26,30,28,26,33,41,32,24,24,34,25,33,29,27,42,31,40,33,28,27,34,27,22,22,27,25,29,29,36,35,47,29,26,29,26,24,25,28,33,25,26,40,26,25,23,29,26,68,49,29,27,25,33,27,31,31,23,38,31,35,36,31,34,42,24,25,26,44,26,30,33,30,29,30,31,24,28,26,25,43,30,27,32,26,25,30,25,30,28,30,31,43,28,27,24,32,29,23,26,25,23,22,31,24,30,27,42,36,23,26,26,28,28,36,34,26,27,36,27,34,29,35,45,22,32,38,32,32,28,36,38,31,27,31,20,25,28,31,31,27,33,37,69,25,28,29,32,28,28,42,35,29,29,37,73,68,49,25,25,31,38,26,26,35,23,40,53,30,37,30,25,30,26,31,59,31,66,32,30,40,28,26,37,23,25,34,44,23,33,32

Per BAM specifications, the 'B:C' type of tags encode an array of unsigned 8-bit integers, i.e. 0 to 255.

Because I need to use the sequenceLayer command on these arrays, I need to load them as BStringSet objects.

What I do for these arrays, where x is the array is:
BStringSet(as(as(as.raw(x),"XRaw"),"BString"))

So the question is if there is an easier / better way to do this conversion.

[hpages]

So the question is if there is an easier / better way to do this conversion.

Not at the moment but the good news is that we finally might have the resources to work on this soon.

[gevro]

That would be great - thanks!

[FelixErnst]

What I do for these arrays, where x is the array is: BStringSet(as(as(as.raw(x),"XRaw"),"BString"))

This will convert it to a character nonetheless instead of retaining the original integer value. In addition you are not covering the standard, since you cannot represent 0 that way

> x <- 0:255
> BStringSet(as(as(as.raw(x),"XRaw"),"BString"))
BStringSet object of length 1:
    width seq
Error in XVector:::extract_character_from_XRaw_by_ranges(x, start, width,  : 
  embedded nul in string: '\0\001\002\003\004\005\006\a\b\t\n\v\f\r\016\017\020\021\022\023\024\025\026\027\030\031\032\033\034\035\036\037 !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJ'

[gevro]

Ok. I'm assuming per @hpages that you all will be able to create a better way to do this soon.

[hpages]

@FelixErnst BStringSet(as(as(as.raw(x),"XRaw"),"BString")) actually does contain the original values:

> bstring <- BStringSet(as(as(as.raw(0:255),"XRaw"),"BString"))
> identical(as.integer(bstring[[1]]), 0:255)
[1] TRUE

In addition you are not covering the standard, since you cannot represent 0 that way

The strings in XString/XStringSet objects are not 0-terminated like C strings. So 0 can actually be part of an XString/XStringSet object. As explained earlier in this thread, the error you are showing above is an issue with the show() method. But the internal representation of the object is correct.

[FelixErnst]

Yes, you are right.

With the conversion the constructor is avoided, which either would call as.character (BStringSet) or would not work at all with integer values (BString). Nice!

• Implement your own XStringQuality. which does not convert any integer values.
• Check that you can still use the BStringSet class as backend to represent the values for your XStringQuality, Otherwise implement your own XStringSet class.
• figure out a meaningful way, how to print those values to console.

So I guess this is a yes for the second point.

[ahl27]

Finally getting around to this--I'm working on adding an optional parameter to display extended ASCII values for BStrings. It would be ideal for to maintain the "1 position = 1 character" display type for other strings, meaning that all values in 0:255 map to exactly one character.

One option to do that is to use some standard set of single character codes that supports values in 0-255. It happens that the Braille character set has exactly 256 characters (encoded in unicode 0x2800-28FF).

This would look like this:

> x <- as(as(as.raw(0:255),"XRaw"),"BString")
> x
256-letter BString object
Error in XVector:::extract_character_from_XRaw_by_ranges(x, start, width,  : 
  embedded nul in string: '\0\001\002\003\004\005\006\a\b\t\n\v\f\r\016\017\020\021\022\023\024\025\026\027\030\031\032\033\034\035\036\037 !"#'
> options(Biostrings.showRaw=TRUE)
> x
256-letter BString object
seq: ⠀⠁⠂⠃⠄⠅⠆⠇⠈⠉⠊⠋⠌⠍⠎⠏⠐⠑⠒⠓⠔⠕⠖⠗⠘⠙⠚⠛⠜⠝⠞⠟⠠⠡⠢⠣...⣜⣝⣞⣟⣠⣡⣢⣣⣤⣥⣦⣧⣨⣩⣪⣫⣬⣭⣮⣯⣰⣱⣲⣳⣴⣵⣶⣷⣸⣹⣺⣻⣼⣽⣾⣿
> 

It's not super readable, but our options are relatively limited when it comes to maintaining readability with a 256 character set...it's at least more readable than the current implementation. Definitely open to other suggestions on that front, for now I'm just looking to have something that can display at all. Most of the BString functionality works perfectly well aside from the printing to the console bit, so it seems like that's the main task to solve here.

Currently working on a more robust version, I'll update with a link when it's in a semi-ready state.